BayXenSmooth - Smooth and Spatial Bayesian Clustering for Xenium Data

Recent advancements in spatially resolved transcriptomics have opened new opportunities to map tissue regions into biologically meaningful clusters. These clusters can be essential for applications such as biomarker discovery and sub-population detection. While spatial transcriptomics data may sometimes include reference labels, such labels often have limitations, and there is a growing need to improve methods in reference-free settings. To address this challenge, we introduce BayXenSmooth - a stochastic variational inference (SVI) method designed to learn posterior spot cluster distributions that are both spatially coherent and biologically interpretable. BayXenSmooth enhances clustering accuracy by incorporating spatial relationships through carefully designed prior distributions, which allows it to balance the trade-off between smoothness and expression differences. Furthermore, the method is scalable and effective across data resolutions. As spot data scales polynomially with finer resolution, BayXenSmooth’s use of SVI makes it more computationally efficient than previous methods relying on Markov Chain Monte Carlo (MCMC), which can be prohibitively expensive to retrain. Additionally, BayXenSmooth supports online updates, enabling dynamic adaptation to new data without the need for complete retraining, a significant advantage over traditional MCMC approaches. Our results demonstrate that BayXenSmooth effectively groups tissues into smoother regions compared to previous methods while preserving expression heterogeneity consistent with earlier studies, offering a competitive alternative to existing approaches.