Multimodal Language Modeling for High-Accuracy Single Cell
  Transcriptomics Analysis and Generation
Pre-trained language models (PLMs) have revolutionized scientific research,
yet their application to single-cell analysis remains limited. Text PLMs cannot
process single-cell RNA sequencing data, while cell PLMs lack the ability to
handle free text, restricting their use in multimodal tasks. Existing efforts
to bridge these modalities often suffer from information loss or inadequate
single-modal pre-training, leading to suboptimal performances. To address these
challenges, we propose Single-Cell MultiModal Generative Pre-trained
Transformer (scMMGPT), a unified PLM for joint cell and text modeling. scMMGPT
effectively integrates the state-of-the-art cell and text PLMs, facilitating
cross-modal knowledge sharing for improved performance. To bridge the text-cell
modality gap, scMMGPT leverages dedicated cross-modal projectors, and undergoes
extensive pre-training on 27 million cells -- the largest dataset for
multimodal cell-text PLMs to date. This large-scale pre-training enables
scMMGPT to excel in joint cell-text tasks, achieving an 84\% relative
improvement of textual discrepancy for cell description generation, 20.5\%
higher accuracy for cell type annotation, and 4\% improvement in k-NN
accuracy for text-conditioned pseudo-cell generation, outperforming baselines.
67d247c2a48964532e40e78e
Yaorui Shi
admin_assigned
2025-03-14T11:26:21.592Z
63edd2d1f765928ceeb49057
https://cdn-avatars.huggingface.co/v1/production/uploads/1676530369930-noauth.png
Yaorui SHI
user
yrshi
67d247c2a48964532e40e78f
Jiaqi Yang
67d247c2a48964532e40e790
Sihang Li
admin_assigned
2025-03-14T11:27:04.999Z
635913a68f38318bfabc90ea
https://cdn-avatars.huggingface.co/v1/production/uploads/635913a68f38318bfabc90ea/29M50hPy2nMwMjMElAEgA.jpeg
Sihang Li
user
Sihangli
67d247c2a48964532e40e791
Junfeng Fang
67d247c2a48964532e40e792
Xiang Wang
67d247c2a48964532e40e793
Zhiyuan Liu
admin_assigned
2025-03-14T11:26:44.031Z
6310a3cd531cc21f9e06de6a
https://cdn-avatars.huggingface.co/v1/production/uploads/6310a3cd531cc21f9e06de6a/aTGMx3O41lUARK9s3dAik.jpeg
Zhiyuan Liu
user
acharkq
67d247c2a48964532e40e794
Yang Zhang
Multimodal Language Modeling for High-Accuracy Single Cell
  Transcriptomics Analysis and Generation
Pre-trained language models (PLMs) have revolutionized scientific research,
yet their application to single-cell analysis remains limited. Text PLMs cannot
process single-cell RNA sequencing data, while cell PLMs lack the ability to
handle free text, restricting their use in multimodal tasks. Existing efforts
to bridge these modalities often suffer from information loss or inadequate
single-modal pre-training, leading to suboptimal performances. To address these
challenges, we propose Single-Cell MultiModal Generative Pre-trained
Transformer (scMMGPT), a unified PLM for joint cell and text modeling. scMMGPT
effectively integrates the state-of-the-art cell and text PLMs, facilitating
cross-modal knowledge sharing for improved performance. To bridge the text-cell
modality gap, scMMGPT leverages dedicated cross-modal projectors, and undergoes
extensive pre-training on 27 million cells -- the largest dataset for
multimodal cell-text PLMs to date. This large-scale pre-training enables
scMMGPT to excel in joint cell-text tasks, achieving an 84\% relative
improvement of textual discrepancy for cell description generation, 20.5\%
higher accuracy for cell type annotation, and 4\% improvement in k-NN
accuracy for text-conditioned pseudo-cell generation, outperforming baselines.
A unified pre-trained language model for joint cell and text analysis improves performance across multimodal tasks by leveraging extensive pre-training and cross-modal projectors.
Pre-trained language models PLMs single-cell RNA sequencing data cross-modal knowledge sharing cross-modal projectors scMMGPT cell description generation cell type annotation k-NN accuracy text-conditioned pseudo-cell generation