scSAMAC: saliency-adjusted masking induced attention contrastive learning for single-cell clustering
Transcriptional activity of genes can explain the unique identities and biological functions of individual cells. Traditional bulk gene expression methods analyze the average transcription levels of bulk cells, ignoring the heterogeneity of individual cells. The rapid development of single-cell sequencing technology has narrowed this gap. Single-cell sequencing provides transcriptomic profiles of individual cells, which are crucial for identifying cell types , studying complex biological systems , and investigating complex diseases . Clustering, as an unsupervised learning method that measures similarity using different distance metrics, is the fundamental approach for scRNA-seq data analysis. However, the high dimensionality, sparsity , and noise events of scRNA-seq data complicate the process of single-cell clustering. Therefore, the development of high-precision cell clustering models is essential. In the past decade, numerous scRNA-seq clustering methods have been proposed. For example, SIMLR  clusters cells using spectral clustering based on similarities between samples. Principal component analysis  performs dimensionality reduction on the original high-dimensional data to cluster them. Seurat  employs the Louvain algorithm to identify clusters on shared nearest-neighbor graphs. ScHFC  is a hybrid fuzzy model optimized through natural computing, combining fuzzy C-means and Gath–Geva to enhance clustering performance. While these methods have indeed improved the performance of single-cell clustering, their limitations have become increasingly evident with the advancement of time. Their clustering performance no longer meets the higher standards demanded by the rapid advancements in single-cell sequencing technology. A higher standard entails more stringent requirements across various clustering evaluation metrics to improve scalability of methods. For single cells, noise and sparsity are common challenges, necessitating modern clustering methods that exhibit robustness to these data characteristics . For instance, some algorithms first perform dimensionality reduction on the raw data, which may lead to the loss of important information. Additionally, when faced with thousands of cells and genes, these methods often require high computational burden, greatly restricting the scalability of single-cell clustering algorithms, thereby affecting downstream analysis. With the advancement of deep learning, the clustering methods on scRNA-seq data have also diversified. Autoencoder is a classical deep neural network model, which designs an encoder to compress the high-dimensional inputs into low-dimensional representations in a latent subspace, and then reconstructs them by a decoder . To extract more efficient features from the latent subspace, many extended versions of autoencoder have been developed, such as denoising autoencoder (DAE)  and variational autoencoder (VAE) . Autoencoder is capable of learning discriminant features from the original inputs, which has also been exploited for scRNA-seq data clustering. Tian et al .  proposed a DAE-based clustering framework, named scDeepCluster, which maps the count matrix of scRNA-seq data with a nonlinear function into the low-dimensional representations by a zero-inflated negative binomial distribution  (ZINB)-based autoencoder, and then performs clustering tasks on the potential space using Kullback–Leibler (KL) divergence. However, these autoencoder-based algorithms solely focus on investigating the data itself, overlooking the inter-cellular relationships, such as pairwise distances between cells. This leads to lower learning efficiency and poorer clustering results. On the other hand, scziDesk  addresses the limitations of models like scDeepCluster by further employing a weighted soft k-means clustering algorithm in the latent space, which forces similar cell types to converge, resulting in superior clustering outcomes. Nonetheless, scziDesk has limitations in handling high-dimensional sparse data and does not improve feature correlation learning for sparse data. To avoid the problem, a novel approach, termed contrast_sc , which considers cell relationships, is presented;it extracts features by contrastively learning the similarity between cells. It first implements representation training and then performs clustering on the embeddings in the second stage. The enhanced representations for each sample are achieved by applying Gaussian noise to the original input data. However, this method still suffers from a classic problem: it overly focuses on the similarity and dissimilarity between samples when learning features, potentially leading to the loss of intrinsic data characteristics. Moreover, randomly adding noise may not be beneficial for extracting important features using the network. In addition, scDGDC  employs triplet contrastive learning to dual graph convolutional networks, achieving excellent clustering performance. NsDCC  proposes a dual-level contrastive learning method for clustering with good results. Besides, some methods enhance clustering performance through feature augmentation. For example, scDFC  proposes both a structure-based and an attention-guided feature learning modules, both of which improve the performance of clustering. scDeepFc  uses a DAE network and a deep graph convolution network to embed high-dimensional gene attribute information and high-order cell–cell topological relation into different low-dimensional representations, and then fuses them to generate a more comprehensive and accurate consensus representation via a deep fusion network. scMFC  develops random walk techniques and a cross-view information aggregation mechanism to better capture higher order relationships of the cell graph. To address the limitations of existing methods, inspired by the aforementioned works, we propose a novel deep framework scSAMAC, shown in , for clustering scRNA-seq data. scSAMAC leverages a saliency-adjusted masking mechanism to generate negative samples for contrastive learning, aiming to capture more distinctive features from scRNA-seq data. Additionally, an attention block is introduced to enhance the intermediate latent features, thereby improving downstream clustering performance. Finally, a novel clustering loss incorporating newly proposed loss functions is employed to further refine clustering outcomes. This framework can resolve the problems related to dropout events and high dimensionality. Meanwhile, it improves the robustness and generalization performance in single-cell clustering. The main contributions of this paper are summarized in three aspects: 1. We construct a mask module, where different masking rates are set for genes based on their expression differences, generating negative samples for contrastive learning. These negative samples are more suitable for VAE framework optimized by negative binomial (NB) loss, preserving significant features for clustering while simulating missing events. Consequently, it enhances clustering performance, robustness, and generalization. 2. We develop a cluster-level loss. By integrating Wasserstein distance and soft k-means loss into it, a hybrid loss is modeled to replace the traditional KL divergence to optimize clustering performance. The cluster-level loss can identify representative features of clusters by leveraging the idea that labels represent the probability of belonging to a specific cluster. Soft k-means loss effectively utilize the pairwise distances between cells. Wasserstein distance is more suitable for sparse data compared with the traditional KL divergence. 3. A block with multi-head attention mechanism is constructed and applied to the latent variables. It extracts feature correlations while retaining some of the original data’s characteristics, preventing excessive feature information loss, which could degrade clustering results. Additionally, the attention block can well utilize existing features to fill in missing data, enabling information integration and repair, thus enhancing data completeness and usability. 

scSAMAC builds a VAE-based deep framework for scRNA-seq data clustering. It consists of a masking module to generate negative samples, two encoders to learn intermediate latent features from the original and masking data, two attention blocks to enhance intermediate latent features via contrastive technique in the pretraining phase, and two contrastive decoders for the final clustering. To optimize the proposed deep network, a contrastive loss is first developed for training, and then a cluster-level loss, combining Wasserstein distance and soft k-means loss, is modeled for clustering. These losses, together with NB or ZINB loss, are used, respectively, in training and clustering to optimize model performance. scSAMAC is divided into two stages: pretraining and clustering. The original input \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$ X $\end{document} and the masked input \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$ X_{{mask}} $\end{document} are separately passed through the encoder, and the resulting latent variables are then fed into an attention block. The latent variable \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$ z $\end{document} , generated from the original input, is then passed to the decoder to compute NB or ZINB loss. In the pretraining phase, the generated \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$ z $\end{document} and \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$ z_{{pair}} $\end{document} are used for contrastive learning to produce the contrast loss. In the clustering phase, instead, \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$ z $\end{document} initially generates \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$ L_{{base}} $\end{document} . Subsequently, \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$ z $\end{document} and \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$ z_{{pair}} $\end{document} are each passed through a decoder to match the number of predicted clusters. The resulting \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$ y $\end{document} and \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$ y_{{mask}} $\end{document} are used for column-wise comparison to produce the cluster-level contrast loss. Preprocessing of scRNA-seq data The preprocessing of raw scRNA-seq read count data is also conducted using the Python library SCANPY . Initially, genes lacking expression in any cell are filtered out. Subsequently, size factors are computed to normalize read counts based on library size, ensuring uniform total counts across cells. Formally, denoting the library size (total read counts) of cell \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$i$\end{document} as \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$s_{i}$\end{document} , the size factor of cell \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$i$\end{document} is calculated as \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$s_{i} / \text{median}(s)$\end{document} . The final step involves logarithmic transformation and scaling of the read counts, facilitating the attainment of unit variance and zero mean in count values. This preprocessed read count matrix serves as the input for our DAE. Denoising autoencoder NB or ZINB is one of the approximation to distribution of scRNA-seq data. They have been applied to extract denoising embeddings and overcome the discrete property and over-dispersion of scRNA-seq dataset: (1) \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
\begin{align*} & {{P}_{NB}}({{X}_{n\times m}}|,)=\frac{\Gamma ({{X}_{n\times m}}+\theta )}{\Gamma ({{X}_{n\times m}}+1)\Gamma (\theta )}\times{{\left(\frac{\theta} {\theta +\mu }\right)}^{\theta }}\times{{\left(\frac{\mu }{\theta +\mu }\right)}^{{{X}_{ij}}}}, \end{align*}\end{document} (2) \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
\begin{align*} & \scriptsize{P_ {ZINB} ( X_ { n \times m } \ | \pi, \mu, \theta ) = \pi \delta_{ 0 } ( X_ { n \times m } ) + ( 1 - \pi ) \times P_ { N B } ( X_ { n \times m } \ | \mu, \theta ),} \end{align*}\end{document} where \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$X$\end{document} denotes the unprocessed raw input data. The mean \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$\mu $\end{document} and dispersion \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$\theta $\end{document} are parameters of distribution, and \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$\pi $\end{document} represents the probability of dropout events in the original counts. The encoder takes \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$X$\end{document} as input, with its output layer generating a reduced-dimensional latent embedding \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$z$\end{document} . Conversely, the decoder sub-network takes \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$z$\end{document} as input and produces reconstructions of the original input \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$X$\end{document} . Both the encoder and decoder are neural networks based on multi-layer perceptron. Subsequently, three distinct fully connected layers are utilized to fit the mean \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$\mu $\end{document} , dispersion \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$\theta $\end{document} , and the dropout \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$\pi $\end{document} : (3) \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
\begin{align*} & Z={{W}_{encoder}}(X+\epsilon), \end{align*}\end{document} (4) \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
\begin{align*} & D={{W}_{decoder}}(Z), \end{align*}\end{document} (5) \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
\begin{align*} & \mu =F_{diag}\left[ exp^{(D{W}_{\mu})} \right], \end{align*}\end{document} (6) \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
\begin{align*} & \theta =exp^{(D{{W}_{\theta }})}, \end{align*}\end{document} (7) \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
\begin{align*} & {\pi = F_{sigmoid} ( D W_ { \pi } ),} \end{align*}\end{document} where \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$\epsilon $\end{document} represents the random Gaussian noise (0 mean and 0.01 variance), \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$F_{diag}$\end{document} refers to the diagonal matrix construction, and \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$F_{sigmoid}$\end{document} is the sigmoid function. Three different fully connected layers are nonlinear activated by diverse ways. The mean \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$\mu $\end{document} and dispersion \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$\theta $\end{document} are nonnegative, hence exponential function is chosen as an activation function. In order to prevent overfitting, the mean \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$\mu $\end{document} is normalized to the same extent with the raw count size for each cell. The dropout probability \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$\pi $\end{document} should be in a reasonable scale (0–1), so a sigmoid is used for standardization. All the \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$W$\end{document} represent the learnable parameters. Finally, the loss function of DAE is the negative log of NB or ZINB likelihood: (8) \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
\begin{align*} & {{L}_{nb}}\left( \mu,\theta \mid X \right)=\log ({{P}_{NB}}({{X}_{n\times m}}\ |\mu,\theta )), \end{align*}\end{document} (9) \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
\begin{align*} & {{L_ { z i n b }} ( \pi, \mu, \theta | X ) = \log ( P_ { Z IN B } ( X_ { n \times m } | \pi, \mu, \theta ) ).} \end{align*}\end{document} Saliency-adjusted masking module In contrastive learning, selecting effective negative samples is essential for enhancing model performance. We introduce a specialized masking strategy named saliency-adjusted to generate these optimal negative samples. In this approach, the standardized input matrix is processed through a masking module to produce negative samples for contrastive learning. The module begins by calculating the column sums of the matrix, where the column with the highest sum is identified as the “most significant” gene column. Based on these sums, we categorize genes into three types: columns with sums closest to the most significant gene are labeled as “significant” gene columns, those with moderately close sums to it as “moderately significant” gene columns, and those with sums farthest from the most significant gene as “least significant” gene columns. We vary masking probabilities across different gene columns. Highly significant gene columns are masked with the lowest probability, minimizing their chances of being set to zero. Columns of moderate significance have a higher probability of zeroing, while the least significant gene columns are masked with the highest probability. The generated negative samples are incorporated during both the pretraining and clustering stages. This saliency-adjusted masking strategy for the input matrix retains attributes critical for clustering and training while enhancing data heterogeneity. By preserving a diverse range of negative samples, this approach helps the model to generalize effectively to unseen data and to mitigate the risk of over-fitting. Furthermore, it also partially addresses the issue of false zeros introduced by dropout events. Attention block Attention mechanism has shown its effectiveness in natural language processing , playing a significant role in extracting features related to sample correlation. Here, we also construct an attention block by combining multi-head self-attention mechanism with residual strategy, which is applied before each latent variable layer of the encoder as follows: (10) \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
\begin{align*} & Q=K=V={{H}^{l}}, \end{align*}\end{document} (11) \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
\begin{align*} & {{Q}_{i}}^{l}=QW_{i}^{{{Q}^{l}}},{{K}_{i}}^{l}=KW_{i}^{{{K}^{l}}},{{V}_{i}}^{l}=VW_{i}^{{{V}^{l}}}, \end{align*}\end{document} where \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$H^{l}$\end{document} represents the input of the \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$l$\end{document} th layer of the latent variables. \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$Q$\end{document} is query, \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$K$\end{document} is key, and \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$V$\end{document} is value, with the \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$l$\end{document} th layer output \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
${{Q}_{i}}^{l}$\end{document} , \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
${{K}_{i}}^{l}$\end{document} , and \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
${{V}_{i}}^{l}$\end{document} , respectively. \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$W_{i}^{{{Q}^{l}}}$\end{document} , \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$W_{i}^{{{K}^{l}}}$\end{document} , and \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$W_{i}^{{{V}^{l}}}$\end{document} are the weight matrices for \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$Q$\end{document} , \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$K$\end{document} , and \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$V$\end{document} . Attention scores are (12) \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
\begin{align*}& {{a_{i}}^{l} = F_{softmax} ( \frac{ Q_ { i }^{l} {K_ { i }^{l}}^ { T }} { \sqrt{ d_ { k }} } ) V_ { i }^{l},}\end{align*}\end{document} where \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$a^{l}$\end{document} is the output head, \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$F_{softmax}$\end{document} represents the softmax function, and \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$d_ { k }$\end{document} is the scaling factor. (13) \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
\begin{align*}& F_{multiHead}({Q^{l}},{K^{l}},{V^{l}}) = F_{concat}( a_ { 1 }^{l},a_ { 2 }^{l}\ldots,{a^{l}}_{M}){W^{l}},\end{align*}\end{document} where \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$W^{l}$\end{document} is the weight matrix used to map the concatenated multi-head output to the final dimension, \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$F_{concat}$\end{document} represents the vector concatenation function, and \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$F_{multiHead}$\end{document} refers to the multi-head attention function. The output should be (14) \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
\begin{align*}& {H_ { attention }^{l}}=F_{multiHead}({{Q}^{l}},{{K}^{l}},{{V}^{l}})+{{H}^{l}},\end{align*}\end{document} where \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$M$\end{document} denotes the number of heads, and \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$H_ { attention }^{l}$\end{document} represents the output enhanced data handled by the attention block. For those high-dimensional data, there may exist redundant or correlated features. The multi-head attention strategy can effectively capture the correlation between them and improve feature representation learning. This is particularly important for sparse data, where many features may have zero values. Therefore, the multi-head attention helps identify features more important for the target task, while the residual mechanism retains some features of the original data to prevent the deterioration of clustering results due to excessive feature losing. Loss function To effectively utilize the information such as between-cell similarity and to address both issues as the asymmetry in traditional clustering processes and the meaningless KL divergence values for distributions, we employ a novel composite loss function here: (15) \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
\begin{align*}& {{L}_{{base}}}=\theta \sum\limits_{i=1}^{n}{\sum\limits_{r=1}^{K}{{{w}_{ir}}}}||{{z}_{i}}-{{\nu }_{r}}|{{|}^{2}}+W_{d}(U,V),\end{align*}\end{document} where \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$z$\end{document} represents low-dimensional latent variables, \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$v_{r}$\end{document} denotes the cluster center in latent space, \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$K$\end{document} is the predicted number of clusters, \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$\theta $\end{document} is a trade-off parameter, \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$W_{d}$\end{document} is Wasserstein distance between the real distribution \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$U$\end{document} and the target distribution \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$V$\end{document} , and \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$w_{ir}$\end{document} is the weight defined as (16) \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
\begin{align*}& {{\tilde{w}}_{ir}}=\frac{e^{(-||{{Z}_{i}}-{{\nu }_{r}}|{{|}^{2}})}}{\sum\limits_{k=1}^{k}{e}^{(-||{{Z}_{i}}-{{\nu }_{k}}|{{|}^{2}})}}.\end{align*}\end{document} To expedite the convergence, we also perform weight inflation to it: (17) \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
\begin{align*}& {{w}_{ir}}=\frac{\tilde{w}_{ij}^{2}}{\sum\limits_{j=1}^{K}{(\tilde{w}_{ij}^{2})}}.\end{align*}\end{document} For Wasserstein distance \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$W_{d}$\end{document} , its advantages are 1. Wasserstein distance delineates the minimal cost of moving from one probability density to another by capturing the differences between distribution patterns effectively. 2. For any two nonoverlapping distributions, Wasserstein distance can still balance their differences, while KL divergence may be undefined outside the function. 3. Wasserstein distance possesses symmetry and the triangle inequality, addressing the issues of asymmetry and non-transitivity in KL divergence. It is expressed as (18) \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
\begin{align*}& W_{d}(U,V)=in{{f}_{\gamma \sim\prod{(U,V)}}}{{E}_{(x,y\sim\gamma )}}\left[ |x-y| \right],\end{align*}\end{document} where inf refers to the largest lower bound, and \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$\gamma \sim (U, V)$\end{document} denotes all possible joint distributions between the distributions \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$U$\end{document} and \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$V$\end{document} . For each possible joint distribution \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$\gamma $\end{document} , \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$(x, y) \sim \gamma $\end{document} means that \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$x$\end{document} and \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$y$\end{document} are sampled from \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$\gamma $\end{document} . \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$E$\end{document} represents the expectation of the sample on the distance \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$|x-y|$\end{document} under the joint distribution \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$\gamma $\end{document} . Contrastive loss in the pretraining phase In image processing, contrastive learning is commonly used to learn image representations in a low-dimensional subspace , causing similar images to be close, while dissimilar images apart. This learning paradigm can be applied to various related tasks. As a result, we extend contrastive learning to single-cell clustering. We apply the aforementioned masking strategy to the input matrix to generate negative samples \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$X_{{mask}}$\end{document} . These negative samples are then fed into a VAE, and the latent variable \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$z_{{pair}}$\end{document} is obtained from the last layer of the encoder. Similarly, the unprocessed input \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$X$\end{document} undergoes the same process to obtain the latent variable \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$z$\end{document} . The contrastive loss is calculated as (19) \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
\begin{align*}& {{L}_{contra}}=\sum\limits_{i=1}^{2N}{-\log (({e^{({{z}_{i}}\cdot{{z}_{pair(i)}}/\tau} )} /{{(\sum\limits_{i=1}^{2N}{{{1}_{i\ne j}}}e^{({{z}_{i}}\cdot{{z}_{p\text{ai}r(i)}}/\tau )}}})}).\end{align*}\end{document} Cluster-level contrastive loss At this stage, we adhere to the principle of “labels as representations.” When projecting data samples into a space with dimensions equal to the number of clusters, the \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$i$\end{document} th element of the feature vector can be interpreted as the probability of the sample belonging to the \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$i$\end{document} th cluster. Consequently, the feature vector represents the soft labels of the sample. Therefore, we apply the generated negative samples and positive samples, as described above, through a VAE to obtain the latent variables \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$z_{{pair}}$\end{document} and \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$z$\end{document} . These latent variables are then fed into the corresponding decoder, where the number of output features equals the predicted number of clusters obtained after pretraining. Consequently, we obtain \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$y_{{mask}}$\end{document} and \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$y$\end{document} , which are used for subsequent clustering. The clustering loss function is then calculated as follows. For each dimension \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$i$\end{document} , we calculate the squared Euclidean distance between the \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$i$\end{document} th columns of \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$y_{{mask}}$\end{document} and \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$y$\end{document} , and obtain their sum: (20) \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
\begin{align*}& {{L}_{pos}}=\sum\limits_{i=1}^{n}{||{{a}_{i}}-{{b}_{i}}|{{|}^{2}}},\end{align*}\end{document} where \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$a_{i}$\end{document} and \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$b_{i}$\end{document} are the \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$i$\end{document} th columns of \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$y_{{mask}}$\end{document} and \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$y$\end{document} , respectively. For all different column pairs \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$(i, j)$\end{document} , we can compute the Euclidean distance between the two columns, using the margin parameter \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$m$\end{document} to adjust the following negative loss: (21) \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
\begin{align*}& {{L}_{neg}}=\sum\limits_{i=1}^{n}{\sum\limits_{j\ne i}{\max}} {{(0,-m||{{a}_{i}}-{{b}_{j}}||)}^{2}}.\end{align*}\end{document} We then combine the losses for both the positive and negative sample pairs with the following formulation as \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
${L}_{clu-lev}$\end{document} : (22) \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
\begin{align*}& {{L}_{clu-lev}}=\frac{1}{{{n}^{2}}}\left( {{L}_{pos}}+{{L}_{neg}} \right).\end{align*}\end{document} Then, the loss for the entire learning process is comprised of the pretraining loss and the clustering loss, which are stated below: (23) \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
\begin{align*} & {{L}_{pret\text{r}ain}}=\mu{{L}_{contra}}+{{L}_{nb}}, \end{align*}\end{document} (24) \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
\begin{align*} & {{L}_{cluster}}={{L}_{base}}+{{L}_{nb}}+\omega{{L}_{clu-lev}}. \end{align*}\end{document} \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
${L}_{zinb}$\end{document} is often adopted to train the AE framework, which is also considered in the proposed method by substituting \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
${L}_{zinb}$\end{document} for \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
${L}_{nb}$\end{document} to make comparisons in the following experiments. 

The preprocessing of raw scRNA-seq read count data is also conducted using the Python library SCANPY . Initially, genes lacking expression in any cell are filtered out. Subsequently, size factors are computed to normalize read counts based on library size, ensuring uniform total counts across cells. Formally, denoting the library size (total read counts) of cell \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$i$\end{document} as \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$s_{i}$\end{document} , the size factor of cell \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$i$\end{document} is calculated as \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$s_{i} / \text{median}(s)$\end{document} . The final step involves logarithmic transformation and scaling of the read counts, facilitating the attainment of unit variance and zero mean in count values. This preprocessed read count matrix serves as the input for our DAE. 

NB or ZINB is one of the approximation to distribution of scRNA-seq data. They have been applied to extract denoising embeddings and overcome the discrete property and over-dispersion of scRNA-seq dataset: (1) \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
\begin{align*} & {{P}_{NB}}({{X}_{n\times m}}|,)=\frac{\Gamma ({{X}_{n\times m}}+\theta )}{\Gamma ({{X}_{n\times m}}+1)\Gamma (\theta )}\times{{\left(\frac{\theta} {\theta +\mu }\right)}^{\theta }}\times{{\left(\frac{\mu }{\theta +\mu }\right)}^{{{X}_{ij}}}}, \end{align*}\end{document} (2) \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
\begin{align*} & \scriptsize{P_ {ZINB} ( X_ { n \times m } \ | \pi, \mu, \theta ) = \pi \delta_{ 0 } ( X_ { n \times m } ) + ( 1 - \pi ) \times P_ { N B } ( X_ { n \times m } \ | \mu, \theta ),} \end{align*}\end{document} where \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$X$\end{document} denotes the unprocessed raw input data. The mean \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$\mu $\end{document} and dispersion \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$\theta $\end{document} are parameters of distribution, and \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$\pi $\end{document} represents the probability of dropout events in the original counts. The encoder takes \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$X$\end{document} as input, with its output layer generating a reduced-dimensional latent embedding \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$z$\end{document} . Conversely, the decoder sub-network takes \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$z$\end{document} as input and produces reconstructions of the original input \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$X$\end{document} . Both the encoder and decoder are neural networks based on multi-layer perceptron. Subsequently, three distinct fully connected layers are utilized to fit the mean \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$\mu $\end{document} , dispersion \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$\theta $\end{document} , and the dropout \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$\pi $\end{document} : (3) \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
\begin{align*} & Z={{W}_{encoder}}(X+\epsilon), \end{align*}\end{document} (4) \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
\begin{align*} & D={{W}_{decoder}}(Z), \end{align*}\end{document} (5) \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
\begin{align*} & \mu =F_{diag}\left[ exp^{(D{W}_{\mu})} \right], \end{align*}\end{document} (6) \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
\begin{align*} & \theta =exp^{(D{{W}_{\theta }})}, \end{align*}\end{document} (7) \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
\begin{align*} & {\pi = F_{sigmoid} ( D W_ { \pi } ),} \end{align*}\end{document} where \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$\epsilon $\end{document} represents the random Gaussian noise (0 mean and 0.01 variance), \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$F_{diag}$\end{document} refers to the diagonal matrix construction, and \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$F_{sigmoid}$\end{document} is the sigmoid function. Three different fully connected layers are nonlinear activated by diverse ways. The mean \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$\mu $\end{document} and dispersion \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$\theta $\end{document} are nonnegative, hence exponential function is chosen as an activation function. In order to prevent overfitting, the mean \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$\mu $\end{document} is normalized to the same extent with the raw count size for each cell. The dropout probability \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$\pi $\end{document} should be in a reasonable scale (0–1), so a sigmoid is used for standardization. All the \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$W$\end{document} represent the learnable parameters. Finally, the loss function of DAE is the negative log of NB or ZINB likelihood: (8) \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
\begin{align*} & {{L}_{nb}}\left( \mu,\theta \mid X \right)=\log ({{P}_{NB}}({{X}_{n\times m}}\ |\mu,\theta )), \end{align*}\end{document} (9) \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
\begin{align*} & {{L_ { z i n b }} ( \pi, \mu, \theta | X ) = \log ( P_ { Z IN B } ( X_ { n \times m } | \pi, \mu, \theta ) ).} \end{align*}\end{document} 

In contrastive learning, selecting effective negative samples is essential for enhancing model performance. We introduce a specialized masking strategy named saliency-adjusted to generate these optimal negative samples. In this approach, the standardized input matrix is processed through a masking module to produce negative samples for contrastive learning. The module begins by calculating the column sums of the matrix, where the column with the highest sum is identified as the “most significant” gene column. Based on these sums, we categorize genes into three types: columns with sums closest to the most significant gene are labeled as “significant” gene columns, those with moderately close sums to it as “moderately significant” gene columns, and those with sums farthest from the most significant gene as “least significant” gene columns. We vary masking probabilities across different gene columns. Highly significant gene columns are masked with the lowest probability, minimizing their chances of being set to zero. Columns of moderate significance have a higher probability of zeroing, while the least significant gene columns are masked with the highest probability. The generated negative samples are incorporated during both the pretraining and clustering stages. This saliency-adjusted masking strategy for the input matrix retains attributes critical for clustering and training while enhancing data heterogeneity. By preserving a diverse range of negative samples, this approach helps the model to generalize effectively to unseen data and to mitigate the risk of over-fitting. Furthermore, it also partially addresses the issue of false zeros introduced by dropout events. 

Attention mechanism has shown its effectiveness in natural language processing , playing a significant role in extracting features related to sample correlation. Here, we also construct an attention block by combining multi-head self-attention mechanism with residual strategy, which is applied before each latent variable layer of the encoder as follows: (10) \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
\begin{align*} & Q=K=V={{H}^{l}}, \end{align*}\end{document} (11) \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
\begin{align*} & {{Q}_{i}}^{l}=QW_{i}^{{{Q}^{l}}},{{K}_{i}}^{l}=KW_{i}^{{{K}^{l}}},{{V}_{i}}^{l}=VW_{i}^{{{V}^{l}}}, \end{align*}\end{document} where \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$H^{l}$\end{document} represents the input of the \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$l$\end{document} th layer of the latent variables. \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$Q$\end{document} is query, \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$K$\end{document} is key, and \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$V$\end{document} is value, with the \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$l$\end{document} th layer output \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
${{Q}_{i}}^{l}$\end{document} , \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
${{K}_{i}}^{l}$\end{document} , and \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
${{V}_{i}}^{l}$\end{document} , respectively. \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$W_{i}^{{{Q}^{l}}}$\end{document} , \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$W_{i}^{{{K}^{l}}}$\end{document} , and \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$W_{i}^{{{V}^{l}}}$\end{document} are the weight matrices for \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$Q$\end{document} , \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$K$\end{document} , and \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$V$\end{document} . Attention scores are (12) \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
\begin{align*}& {{a_{i}}^{l} = F_{softmax} ( \frac{ Q_ { i }^{l} {K_ { i }^{l}}^ { T }} { \sqrt{ d_ { k }} } ) V_ { i }^{l},}\end{align*}\end{document} where \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$a^{l}$\end{document} is the output head, \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$F_{softmax}$\end{document} represents the softmax function, and \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$d_ { k }$\end{document} is the scaling factor. (13) \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
\begin{align*}& F_{multiHead}({Q^{l}},{K^{l}},{V^{l}}) = F_{concat}( a_ { 1 }^{l},a_ { 2 }^{l}\ldots,{a^{l}}_{M}){W^{l}},\end{align*}\end{document} where \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$W^{l}$\end{document} is the weight matrix used to map the concatenated multi-head output to the final dimension, \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$F_{concat}$\end{document} represents the vector concatenation function, and \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$F_{multiHead}$\end{document} refers to the multi-head attention function. The output should be (14) \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
\begin{align*}& {H_ { attention }^{l}}=F_{multiHead}({{Q}^{l}},{{K}^{l}},{{V}^{l}})+{{H}^{l}},\end{align*}\end{document} where \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$M$\end{document} denotes the number of heads, and \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$H_ { attention }^{l}$\end{document} represents the output enhanced data handled by the attention block. For those high-dimensional data, there may exist redundant or correlated features. The multi-head attention strategy can effectively capture the correlation between them and improve feature representation learning. This is particularly important for sparse data, where many features may have zero values. Therefore, the multi-head attention helps identify features more important for the target task, while the residual mechanism retains some features of the original data to prevent the deterioration of clustering results due to excessive feature losing. 

To effectively utilize the information such as between-cell similarity and to address both issues as the asymmetry in traditional clustering processes and the meaningless KL divergence values for distributions, we employ a novel composite loss function here: (15) \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
\begin{align*}& {{L}_{{base}}}=\theta \sum\limits_{i=1}^{n}{\sum\limits_{r=1}^{K}{{{w}_{ir}}}}||{{z}_{i}}-{{\nu }_{r}}|{{|}^{2}}+W_{d}(U,V),\end{align*}\end{document} where \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$z$\end{document} represents low-dimensional latent variables, \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$v_{r}$\end{document} denotes the cluster center in latent space, \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$K$\end{document} is the predicted number of clusters, \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$\theta $\end{document} is a trade-off parameter, \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$W_{d}$\end{document} is Wasserstein distance between the real distribution \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$U$\end{document} and the target distribution \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$V$\end{document} , and \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$w_{ir}$\end{document} is the weight defined as (16) \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
\begin{align*}& {{\tilde{w}}_{ir}}=\frac{e^{(-||{{Z}_{i}}-{{\nu }_{r}}|{{|}^{2}})}}{\sum\limits_{k=1}^{k}{e}^{(-||{{Z}_{i}}-{{\nu }_{k}}|{{|}^{2}})}}.\end{align*}\end{document} To expedite the convergence, we also perform weight inflation to it: (17) \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
\begin{align*}& {{w}_{ir}}=\frac{\tilde{w}_{ij}^{2}}{\sum\limits_{j=1}^{K}{(\tilde{w}_{ij}^{2})}}.\end{align*}\end{document} For Wasserstein distance \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$W_{d}$\end{document} , its advantages are 1. Wasserstein distance delineates the minimal cost of moving from one probability density to another by capturing the differences between distribution patterns effectively. 2. For any two nonoverlapping distributions, Wasserstein distance can still balance their differences, while KL divergence may be undefined outside the function. 3. Wasserstein distance possesses symmetry and the triangle inequality, addressing the issues of asymmetry and non-transitivity in KL divergence. It is expressed as (18) \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
\begin{align*}& W_{d}(U,V)=in{{f}_{\gamma \sim\prod{(U,V)}}}{{E}_{(x,y\sim\gamma )}}\left[ |x-y| \right],\end{align*}\end{document} where inf refers to the largest lower bound, and \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$\gamma \sim (U, V)$\end{document} denotes all possible joint distributions between the distributions \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$U$\end{document} and \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$V$\end{document} . For each possible joint distribution \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$\gamma $\end{document} , \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$(x, y) \sim \gamma $\end{document} means that \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$x$\end{document} and \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$y$\end{document} are sampled from \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$\gamma $\end{document} . \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$E$\end{document} represents the expectation of the sample on the distance \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$|x-y|$\end{document} under the joint distribution \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$\gamma $\end{document} . 

In image processing, contrastive learning is commonly used to learn image representations in a low-dimensional subspace , causing similar images to be close, while dissimilar images apart. This learning paradigm can be applied to various related tasks. As a result, we extend contrastive learning to single-cell clustering. We apply the aforementioned masking strategy to the input matrix to generate negative samples \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$X_{{mask}}$\end{document} . These negative samples are then fed into a VAE, and the latent variable \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$z_{{pair}}$\end{document} is obtained from the last layer of the encoder. Similarly, the unprocessed input \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$X$\end{document} undergoes the same process to obtain the latent variable \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$z$\end{document} . The contrastive loss is calculated as (19) \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
\begin{align*}& {{L}_{contra}}=\sum\limits_{i=1}^{2N}{-\log (({e^{({{z}_{i}}\cdot{{z}_{pair(i)}}/\tau} )} /{{(\sum\limits_{i=1}^{2N}{{{1}_{i\ne j}}}e^{({{z}_{i}}\cdot{{z}_{p\text{ai}r(i)}}/\tau )}}})}).\end{align*}\end{document} 

At this stage, we adhere to the principle of “labels as representations.” When projecting data samples into a space with dimensions equal to the number of clusters, the \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$i$\end{document} th element of the feature vector can be interpreted as the probability of the sample belonging to the \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$i$\end{document} th cluster. Consequently, the feature vector represents the soft labels of the sample. Therefore, we apply the generated negative samples and positive samples, as described above, through a VAE to obtain the latent variables \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$z_{{pair}}$\end{document} and \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$z$\end{document} . These latent variables are then fed into the corresponding decoder, where the number of output features equals the predicted number of clusters obtained after pretraining. Consequently, we obtain \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$y_{{mask}}$\end{document} and \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$y$\end{document} , which are used for subsequent clustering. The clustering loss function is then calculated as follows. For each dimension \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$i$\end{document} , we calculate the squared Euclidean distance between the \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$i$\end{document} th columns of \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$y_{{mask}}$\end{document} and \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$y$\end{document} , and obtain their sum: (20) \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
\begin{align*}& {{L}_{pos}}=\sum\limits_{i=1}^{n}{||{{a}_{i}}-{{b}_{i}}|{{|}^{2}}},\end{align*}\end{document} where \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$a_{i}$\end{document} and \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$b_{i}$\end{document} are the \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$i$\end{document} th columns of \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$y_{{mask}}$\end{document} and \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$y$\end{document} , respectively. For all different column pairs \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$(i, j)$\end{document} , we can compute the Euclidean distance between the two columns, using the margin parameter \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$m$\end{document} to adjust the following negative loss: (21) \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
\begin{align*}& {{L}_{neg}}=\sum\limits_{i=1}^{n}{\sum\limits_{j\ne i}{\max}} {{(0,-m||{{a}_{i}}-{{b}_{j}}||)}^{2}}.\end{align*}\end{document} We then combine the losses for both the positive and negative sample pairs with the following formulation as \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
${L}_{clu-lev}$\end{document} : (22) \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
\begin{align*}& {{L}_{clu-lev}}=\frac{1}{{{n}^{2}}}\left( {{L}_{pos}}+{{L}_{neg}} \right).\end{align*}\end{document} Then, the loss for the entire learning process is comprised of the pretraining loss and the clustering loss, which are stated below: (23) \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
\begin{align*} & {{L}_{pret\text{r}ain}}=\mu{{L}_{contra}}+{{L}_{nb}}, \end{align*}\end{document} (24) \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
\begin{align*} & {{L}_{cluster}}={{L}_{base}}+{{L}_{nb}}+\omega{{L}_{clu-lev}}. \end{align*}\end{document} \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
${L}_{zinb}$\end{document} is often adopted to train the AE framework, which is also considered in the proposed method by substituting \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
${L}_{zinb}$\end{document} for \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
${L}_{nb}$\end{document} to make comparisons in the following experiments. 

Datasets In experiments, we used 10 single-cell datasets for clustering, as detailed in . These datasets cover a wide range of single-cell data types, allowing for a comprehensive evaluation of the clustering performance of the proposed method. It includes both small-scale and large-scale cell counts, data with a higher number of genes, and multi-category data. Especially, a very large dataset with 27 499 cells and 13 166 genes, i.e. Shekhar_Mouse_Retina, is also introduced for experiments. Additionally, these datasets are annotated with ground truth labels to evaluate the clustering performance. Evaluation metrics All clustering results are measured by Adjusted Rand Index (ARI) , Normalized Mutual Information (NMI), and Accuracy (CA). Given the two clustering assignments \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$U$\end{document} and \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$V$\end{document} on a set with \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$n$\end{document} data points, which have \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$C_{U}$\end{document} and \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$C_{V}$\end{document} clusters, respectively, NMI is defined as the mutual information between \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$U$\end{document} and \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$V$\end{document} divided by the entropy of the clustering \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$U$\end{document} and \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$V$\end{document} . Specifically, (25) \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
\begin{align*}& NMI=\frac{\sum\nolimits_{\text{p}=1}^{{{C}_{U}}}{\sum\nolimits_{q=1}^{{{C}_{V}}}{|{{U}_{p}}\cap{{V}_{q}}|}}\log \frac{n|{{U}_{p}}\cap{{V}_{q}}|}{|{{U}_{p}}|\times |{{V}_{q}}|}}{\max (-\sum\limits_{p=1}^{{{C}_{U}}}{|{{U}_{p}}|\log \frac{|{{U}_{p}}|}{n}},-\sum\limits_{q=1}^{{{C}_{V}}}{|{{V}_{q}}|\log \frac{|{{V}_{q}}|}{n}})}.\end{align*}\end{document} CA is defined as the best matching between a cluster assignment and the ground truth assignment. Given a data point \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$i$\end{document} , let \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$l_{i}$\end{document} be the ground truth label and \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$u_{i}$\end{document} be the assignment of the clustering algorithm; CA is then defined as (26) \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
\begin{align*}& CA=\underset{m}{\mathop{\max}}\,\frac{\sum\limits_{i=1}^{n}{1}\left\{ {{l}_{i}}=m({{u}_{i}}) \right\}}{n},\end{align*}\end{document} where \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$n$\end{document} is the number of data points, and \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$m$\end{document} ranges over all possible one-to-one mappings between cluster assignments and true labels. The best mapping can be efficiently found using the Hungarian algorithm . Rand Index (RI)  is a simple measure of agreement between two cluster assignments \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$U$\end{document} and \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$V$\end{document} . ARI corrects it for the lack of a constant value of RI when the cluster assignments are selected randomly . ARI is calculated using four quantities. Specifically, we define: 1. The number of pairs of two objects of the same group in both \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$U$\end{document} and \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$V$\end{document} : \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$a$\end{document} 2. The number of pairs of two objects of different groups in both \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$U$\end{document} and \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$V$\end{document} : \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$b$\end{document} 3. The number of pairs of two objects of the same group in \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$U$\end{document} but in different groups in \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$V$\end{document} : \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$c$\end{document} 4. The number of pairs of two objects of different groups in \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$U$\end{document} but in the same group in \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$V$\end{document} : \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$d$\end{document} ARI is formally defined as (27) \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
\begin{align*}& ARI=\frac{\left( \begin{matrix} n \\ 2 \\ \end{matrix} \right)(a+d)-[(a+b)(a+c)+(c+d)(b+d)]}{\left( \begin{matrix} n \\ 2 \\ \end{matrix} \right)-[(a+b)(a+c)+(c+d)(b+d)]}.\end{align*}\end{document} Results To justify the effectiveness of our proposed method, we compared it with eight state-of-the-art methods: scNAME , scDCC , scDeepCluster, contrast_sc, scDSSC , scDCCA , scTPC , and scGMMAE . We also evaluated the results of scSAMAC using NB loss, as well as ZINB loss under the condition that other factors remain unchanged. We used ARI, NMI, and CA to assess their performance, where the higher scores indicate better performance. The performance results across 10 datasets for each method are presented in , , and . The visualizations on two datasets by t-SNE are shown in . Additionally, ARI, NMI, and CA are illustrated in with bars and boxes, respectively. As shown in , , and , it is evident that the proposed method, scSAMAC, generally outperforms the other eight methods across 10 datasets: scSAMAC consistently achieves the highest ARI values in nearly all 10 datasets. scSAMAC demonstrates comparatively better performance on CA, achieving the highest scores across all methods except for scTPC on CITE_CBMC. Although scSAMAC shows a relatively lower NMI value on QS_seq2_Trachea, human2, and human4, it maintains high scores in ARI and CA for these data sets and performs well in NMI on other datasets. In , the results after visualization demonstrate a better clustering performance of scSAMAC than those of other comparison methods. In addition, as shown in , and , scSAMAC is also superior to other comparison methods. , and display that scSAMAC significantly outperforms other clustering methods on certain datasets, particularly showing substantial advantages over all three evaluation metrics on zeisel dataset. Despite exhibiting lower scores in specific metrics for a few datasets, scSAMAC overall shows the best stability in clustering performance. For instance, scSAMAC displays the smallest range between upper and lower limits and the least fluctuation in CA. The median values (indicated by the red lines in the box plots) across the three evaluation criteria further emphasize the superiority of scSAMAC. As shown in , , and , the clustering results of scSAMAC using NB loss slightly underperform those of scSAMAC with ZINB loss by <1% on two clustering metrics for the Shekhar_Mouse_Retina dataset. However, on the other nine datasets, scSAMAC with NB loss consistently outperforms scSAMAC with ZINB loss across all three evaluation metrics. Running time Moreover, we compare the runtime of the scSAMAC with other methods, all executed on an RTX 4090 GPU (24GB), supported by a 16 vCPU Intel(R) Xeon(R) Platinum 8352V CPU @ 2.10GHz, 120GB of RAM, and running on an Ubuntu 18.04.5 LTS operating system, as shown in . It reveals that scSAMAC, along with other deep learning-based methods such as scDeepCluster, does not demonstrate a runtime advantage over the contrast_sc algorithm. Moreover, scSAMAC requires more time than both scDeepCluster and scDSSC. The increased runtime in scSAMAC is attributed to the computational demands of the masking strategy and negative sample generation, both of which require the longer processing time. At the same time, scSAMAC with NB loss not only shows advantages in clustering results compared with scSAMAC with ZINB loss, but it is also slightly faster. However, leveraging optimized hardware or alternative strategies could potentially reduce the runtime of deep learning methods, as a promising direction for future improvements. Parameters setting There are some parameters involved in the proposed scSAMAC algorithm. Thus, in the experiments, they are set via parameter sensitivity analysis. VAE module For the whole VAE, we partially referred to some parameters as scDeepCluster. For instance, we set the training epochs to 400 during the pretraining process, and the learning rate to 1 during the clustering stage. However, to accommodate the contrastive learning loss, we adjusted the learning rate for pretraining to 0.0001. Attention block In the attention module, we set the number of heads for the multi-head attention mechanism to 8 for both the latent variables with dimensions of 64 and 32, yielding the best results. Masking module In generating contrastive phase, we adopted a saliency-adjusted masking strategy. Specifically, we labeled the gene columns with the highest column sum as the most salient gene columns. Columns with sums less than one-tenth of the most salient column were labeled as non-salient columns. Columns with sums between one-tenth and three-tenths of the most salient column were labeled as moderately salient columns, and the remaining columns were labeled as salient columns. Subsequently, we zeroed out some columns based on their saliency: 10% probability for salient columns, 30% for moderately salient columns, and 60% for non-salient columns. All these parameters are adjustable. Since we introduced contrastive loss alongside NB loss during the pretraining, we designed a hyper-parameter \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$\mu $\end{document} to balance their relationship. Through a series of parameter sensitivity experiments (refer to and ), we set it to 0.001. In the clustering phase, we set the hyperparameter \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$ \omega $\end{document} for the cluster-level loss to 0.01 (refer to and ). Convergence To further observe the convergence of the scSAMAC framework, we plotted the total loss during the clustering stage on mouse_ES_cell and QS_seq2_Trachea datasets. As shown in , it can be seen that with the increasing of iterations, the proposed model gradually converges and maintains stability with minor fluctuations. Ablation study To evaluate the contributions of the newly added modules in scSAMAC and to measure their effectiveness, we conducted ablation experiments on the saliency-adjusted masking module used for contrastive loss and the cluster-level loss module. We compared ARI, NMI, and CA before and after the ablation, as shown in , and . It is evident that these two modules contribute to improving clustering performance. Methods retaining the new modules demonstrate higher ARI, NMI, and CA values compared with those by removing the corresponding modules. Furthermore, it is clear that the contrastive loss with the new masking module significantly enhances clustering performance. Differential expression analysis Differential expression (DE) is key to analyzing scRNA-seq data. In this section, we adopted klein dataset to evaluate the performance of scSAMAC for DE. Specifically, scSAMAC employed the Wilcoxon test to detect potential marker genes for each cluster, where the top 20 marker genes were selected and shown in . To visualize the expression levels of the top-ranked marker genes within each cluster, we used several plotting methods: violin plots ( ) illustrate the distribution of marker gene expression; bubble plots ( ) show the proportion of cells expressing the top three marker genes; and heatmaps ( ) depict the expression levels of the top three marker genes across different clusters. These visualizations help to observe the proportion and distribution of marker gene expression in different clusters, providing a comprehensive view of the DE patterns in scRNA-seq data. From the visualization of marker genes in each cluster, it is evident that each cell cluster has specific marker genes. These marker genes can be used to effectively cluster cells. Notably, the marker genes for each cluster are almost unique, strongly supporting our masking strategy. This uniqueness enhances the possibility of applying targeted masking techniques to preserve significant features during training, which improves the overall clustering performance and robustness of our model. Dropout-robust analysis To evaluate scSAMAC’s ability to handle dropout events, we selected two standard dataset, klein and mouse_ES_cell, applying masking rates of 0%, 5%, 10%, 15%, 20%, and 25% to simulate the impact of dropout events on them. The clustering results after applying these dropout rates are shown in . It is evident that, for both datasets, scSAMAC’s clustering performance declines with the increasing of dropout rates. However, when the dropout rate remains below 20%, the decline is not significant, indicating that scSAMAC effectively manages low occurring dropout events. 

In experiments, we used 10 single-cell datasets for clustering, as detailed in . These datasets cover a wide range of single-cell data types, allowing for a comprehensive evaluation of the clustering performance of the proposed method. It includes both small-scale and large-scale cell counts, data with a higher number of genes, and multi-category data. Especially, a very large dataset with 27 499 cells and 13 166 genes, i.e. Shekhar_Mouse_Retina, is also introduced for experiments. Additionally, these datasets are annotated with ground truth labels to evaluate the clustering performance. 

All clustering results are measured by Adjusted Rand Index (ARI) , Normalized Mutual Information (NMI), and Accuracy (CA). Given the two clustering assignments \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$U$\end{document} and \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$V$\end{document} on a set with \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$n$\end{document} data points, which have \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$C_{U}$\end{document} and \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$C_{V}$\end{document} clusters, respectively, NMI is defined as the mutual information between \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$U$\end{document} and \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$V$\end{document} divided by the entropy of the clustering \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$U$\end{document} and \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$V$\end{document} . Specifically, (25) \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
\begin{align*}& NMI=\frac{\sum\nolimits_{\text{p}=1}^{{{C}_{U}}}{\sum\nolimits_{q=1}^{{{C}_{V}}}{|{{U}_{p}}\cap{{V}_{q}}|}}\log \frac{n|{{U}_{p}}\cap{{V}_{q}}|}{|{{U}_{p}}|\times |{{V}_{q}}|}}{\max (-\sum\limits_{p=1}^{{{C}_{U}}}{|{{U}_{p}}|\log \frac{|{{U}_{p}}|}{n}},-\sum\limits_{q=1}^{{{C}_{V}}}{|{{V}_{q}}|\log \frac{|{{V}_{q}}|}{n}})}.\end{align*}\end{document} CA is defined as the best matching between a cluster assignment and the ground truth assignment. Given a data point \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$i$\end{document} , let \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$l_{i}$\end{document} be the ground truth label and \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$u_{i}$\end{document} be the assignment of the clustering algorithm; CA is then defined as (26) \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
\begin{align*}& CA=\underset{m}{\mathop{\max}}\,\frac{\sum\limits_{i=1}^{n}{1}\left\{ {{l}_{i}}=m({{u}_{i}}) \right\}}{n},\end{align*}\end{document} where \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$n$\end{document} is the number of data points, and \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$m$\end{document} ranges over all possible one-to-one mappings between cluster assignments and true labels. The best mapping can be efficiently found using the Hungarian algorithm . Rand Index (RI)  is a simple measure of agreement between two cluster assignments \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$U$\end{document} and \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$V$\end{document} . ARI corrects it for the lack of a constant value of RI when the cluster assignments are selected randomly . ARI is calculated using four quantities. Specifically, we define: 1. The number of pairs of two objects of the same group in both \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$U$\end{document} and \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$V$\end{document} : \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$a$\end{document} 2. The number of pairs of two objects of different groups in both \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$U$\end{document} and \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$V$\end{document} : \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$b$\end{document} 3. The number of pairs of two objects of the same group in \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$U$\end{document} but in different groups in \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$V$\end{document} : \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$c$\end{document} 4. The number of pairs of two objects of different groups in \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$U$\end{document} but in the same group in \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$V$\end{document} : \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$d$\end{document} ARI is formally defined as (27) \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
\begin{align*}& ARI=\frac{\left( \begin{matrix} n \\ 2 \\ \end{matrix} \right)(a+d)-[(a+b)(a+c)+(c+d)(b+d)]}{\left( \begin{matrix} n \\ 2 \\ \end{matrix} \right)-[(a+b)(a+c)+(c+d)(b+d)]}.\end{align*}\end{document} 

To justify the effectiveness of our proposed method, we compared it with eight state-of-the-art methods: scNAME , scDCC , scDeepCluster, contrast_sc, scDSSC , scDCCA , scTPC , and scGMMAE . We also evaluated the results of scSAMAC using NB loss, as well as ZINB loss under the condition that other factors remain unchanged. We used ARI, NMI, and CA to assess their performance, where the higher scores indicate better performance. The performance results across 10 datasets for each method are presented in , , and . The visualizations on two datasets by t-SNE are shown in . Additionally, ARI, NMI, and CA are illustrated in with bars and boxes, respectively. As shown in , , and , it is evident that the proposed method, scSAMAC, generally outperforms the other eight methods across 10 datasets: scSAMAC consistently achieves the highest ARI values in nearly all 10 datasets. scSAMAC demonstrates comparatively better performance on CA, achieving the highest scores across all methods except for scTPC on CITE_CBMC. Although scSAMAC shows a relatively lower NMI value on QS_seq2_Trachea, human2, and human4, it maintains high scores in ARI and CA for these data sets and performs well in NMI on other datasets. In , the results after visualization demonstrate a better clustering performance of scSAMAC than those of other comparison methods. In addition, as shown in , and , scSAMAC is also superior to other comparison methods. , and display that scSAMAC significantly outperforms other clustering methods on certain datasets, particularly showing substantial advantages over all three evaluation metrics on zeisel dataset. Despite exhibiting lower scores in specific metrics for a few datasets, scSAMAC overall shows the best stability in clustering performance. For instance, scSAMAC displays the smallest range between upper and lower limits and the least fluctuation in CA. The median values (indicated by the red lines in the box plots) across the three evaluation criteria further emphasize the superiority of scSAMAC. As shown in , , and , the clustering results of scSAMAC using NB loss slightly underperform those of scSAMAC with ZINB loss by <1% on two clustering metrics for the Shekhar_Mouse_Retina dataset. However, on the other nine datasets, scSAMAC with NB loss consistently outperforms scSAMAC with ZINB loss across all three evaluation metrics. 

Moreover, we compare the runtime of the scSAMAC with other methods, all executed on an RTX 4090 GPU (24GB), supported by a 16 vCPU Intel(R) Xeon(R) Platinum 8352V CPU @ 2.10GHz, 120GB of RAM, and running on an Ubuntu 18.04.5 LTS operating system, as shown in . It reveals that scSAMAC, along with other deep learning-based methods such as scDeepCluster, does not demonstrate a runtime advantage over the contrast_sc algorithm. Moreover, scSAMAC requires more time than both scDeepCluster and scDSSC. The increased runtime in scSAMAC is attributed to the computational demands of the masking strategy and negative sample generation, both of which require the longer processing time. At the same time, scSAMAC with NB loss not only shows advantages in clustering results compared with scSAMAC with ZINB loss, but it is also slightly faster. However, leveraging optimized hardware or alternative strategies could potentially reduce the runtime of deep learning methods, as a promising direction for future improvements. 

There are some parameters involved in the proposed scSAMAC algorithm. Thus, in the experiments, they are set via parameter sensitivity analysis. VAE module For the whole VAE, we partially referred to some parameters as scDeepCluster. For instance, we set the training epochs to 400 during the pretraining process, and the learning rate to 1 during the clustering stage. However, to accommodate the contrastive learning loss, we adjusted the learning rate for pretraining to 0.0001. Attention block In the attention module, we set the number of heads for the multi-head attention mechanism to 8 for both the latent variables with dimensions of 64 and 32, yielding the best results. Masking module In generating contrastive phase, we adopted a saliency-adjusted masking strategy. Specifically, we labeled the gene columns with the highest column sum as the most salient gene columns. Columns with sums less than one-tenth of the most salient column were labeled as non-salient columns. Columns with sums between one-tenth and three-tenths of the most salient column were labeled as moderately salient columns, and the remaining columns were labeled as salient columns. Subsequently, we zeroed out some columns based on their saliency: 10% probability for salient columns, 30% for moderately salient columns, and 60% for non-salient columns. All these parameters are adjustable. Since we introduced contrastive loss alongside NB loss during the pretraining, we designed a hyper-parameter \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$\mu $\end{document} to balance their relationship. Through a series of parameter sensitivity experiments (refer to and ), we set it to 0.001. In the clustering phase, we set the hyperparameter \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$ \omega $\end{document} for the cluster-level loss to 0.01 (refer to and ). 

For the whole VAE, we partially referred to some parameters as scDeepCluster. For instance, we set the training epochs to 400 during the pretraining process, and the learning rate to 1 during the clustering stage. However, to accommodate the contrastive learning loss, we adjusted the learning rate for pretraining to 0.0001. 

In the attention module, we set the number of heads for the multi-head attention mechanism to 8 for both the latent variables with dimensions of 64 and 32, yielding the best results. 

In generating contrastive phase, we adopted a saliency-adjusted masking strategy. Specifically, we labeled the gene columns with the highest column sum as the most salient gene columns. Columns with sums less than one-tenth of the most salient column were labeled as non-salient columns. Columns with sums between one-tenth and three-tenths of the most salient column were labeled as moderately salient columns, and the remaining columns were labeled as salient columns. Subsequently, we zeroed out some columns based on their saliency: 10% probability for salient columns, 30% for moderately salient columns, and 60% for non-salient columns. All these parameters are adjustable. Since we introduced contrastive loss alongside NB loss during the pretraining, we designed a hyper-parameter \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$\mu $\end{document} to balance their relationship. Through a series of parameter sensitivity experiments (refer to and ), we set it to 0.001. In the clustering phase, we set the hyperparameter \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
$ \omega $\end{document} for the cluster-level loss to 0.01 (refer to and ). 

To further observe the convergence of the scSAMAC framework, we plotted the total loss during the clustering stage on mouse_ES_cell and QS_seq2_Trachea datasets. As shown in , it can be seen that with the increasing of iterations, the proposed model gradually converges and maintains stability with minor fluctuations. 

To evaluate the contributions of the newly added modules in scSAMAC and to measure their effectiveness, we conducted ablation experiments on the saliency-adjusted masking module used for contrastive loss and the cluster-level loss module. We compared ARI, NMI, and CA before and after the ablation, as shown in , and . It is evident that these two modules contribute to improving clustering performance. Methods retaining the new modules demonstrate higher ARI, NMI, and CA values compared with those by removing the corresponding modules. Furthermore, it is clear that the contrastive loss with the new masking module significantly enhances clustering performance. 

Differential expression (DE) is key to analyzing scRNA-seq data. In this section, we adopted klein dataset to evaluate the performance of scSAMAC for DE. Specifically, scSAMAC employed the Wilcoxon test to detect potential marker genes for each cluster, where the top 20 marker genes were selected and shown in . To visualize the expression levels of the top-ranked marker genes within each cluster, we used several plotting methods: violin plots ( ) illustrate the distribution of marker gene expression; bubble plots ( ) show the proportion of cells expressing the top three marker genes; and heatmaps ( ) depict the expression levels of the top three marker genes across different clusters. These visualizations help to observe the proportion and distribution of marker gene expression in different clusters, providing a comprehensive view of the DE patterns in scRNA-seq data. From the visualization of marker genes in each cluster, it is evident that each cell cluster has specific marker genes. These marker genes can be used to effectively cluster cells. Notably, the marker genes for each cluster are almost unique, strongly supporting our masking strategy. This uniqueness enhances the possibility of applying targeted masking techniques to preserve significant features during training, which improves the overall clustering performance and robustness of our model. 

To evaluate scSAMAC’s ability to handle dropout events, we selected two standard dataset, klein and mouse_ES_cell, applying masking rates of 0%, 5%, 10%, 15%, 20%, and 25% to simulate the impact of dropout events on them. The clustering results after applying these dropout rates are shown in . It is evident that, for both datasets, scSAMAC’s clustering performance declines with the increasing of dropout rates. However, when the dropout rate remains below 20%, the decline is not significant, indicating that scSAMAC effectively manages low occurring dropout events. 

To achieve better single-cell clustering performance, this study introduces a novel VAE model for clustering, termed scSAMAC, which employs a new masking method for contrastive learning. In both the pretraining and clustering phases, we use the saliency-adjusted masking module to generate negative samples and conduct contrastive learning during pretraining, combined with NB or ZINB loss. To further optimize clustering performance, we replace the traditional KL divergence with soft k-means loss, the Wasserstein distance, and a newly proposed cluster-level loss in the clustering phase. Additionally, we incorporate an attention module in the latent variable layers of the encoder to well extract features from scRNA-seq data. Experiments on several benchmark datasets with other state-of-the-art algorithms demonstrate the high clustering performance of the proposed algorithm, where ARI, NMI, and CA are used to evaluate clustering effectiveness. Our masking method implements a dynamic masking strategy, adapting to various single-cell data. The introduction of attention blocks enhances data integrity and improves data correlation learning, along with new loss functions, enabling the handling of more complex single-cell data. Thus, we believe that the proposed method will be beneficial for future single-cell clustering applications. However, scSAMAC model still has some limitations. Firstly, the generation of negative samples involves a high degree of randomness, requiring multiple experiments to achieve ideal results. Additionally, we often observe that the initial clustering results generated after pretraining are quite good, but improvements during the clustering phase are limited or even decline, which demands more loss functions to optimize the clustering phase and enhance clustering results. Furthermore, although scSAMAC demonstrated reasonable effectiveness on single-cell data, it is time-consuming and less stable when compared with methods like SHARP  and DUBStepR , which excel at fast and efficient clustering in datasets with millions of samples. scSAMAC, as a deep learning method, has disadvantages in computational burden compared with shallow methods like SHARP and DUBStepR, and it faces scalability challenges due to its time-consuming negative sample generation within the masking module of the VAE framework, coupled with the contrastive learning technique. Moreover, the negative sample generation during the masking process may be less stable for large datasets, further limiting scSAMAC’s extensibility. Consequently, further study on deep learning models, including scSAMAC, is necessary to successfully leverage scRNA-seq data to advance human health. Key Points A deep framework is constructed for single-cell sequence data clustering, where a VAE module, a masking module, and an attention block contrastive learning module are contained. In the mask module, different masking rates are set for genes based on their expression differences, generating negative samples for contrastive learning. An attention block contrastive learning module is applied to the latent variables, by which the feature correlations can be extracted while retaining some of the original data’s characteristics and preventing excessive feature losing that could degrade clustering results. We replace the traditional KL divergence with the Wasserstein distance, a soft k-means loss, and a cluster-level contrastive loss function for clustering. 

A deep framework is constructed for single-cell sequence data clustering, where a VAE module, a masking module, and an attention block contrastive learning module are contained. In the mask module, different masking rates are set for genes based on their expression differences, generating negative samples for contrastive learning. An attention block contrastive learning module is applied to the latent variables, by which the feature correlations can be extracted while retaining some of the original data’s characteristics and preventing excessive feature losing that could degrade clustering results. We replace the traditional KL divergence with the Wasserstein distance, a soft k-means loss, and a cluster-level contrastive loss function for clustering. 

