title: Multimodal Language Modeling for High-Accuracy Single Cell
    Transcriptomics Analysis and Generation
summary: Pre-trained language models (PLMs) have revolutionized scientific research,
  yet their application to single-cell analysis remains limited. Text PLMs cannot
  process single-cell RNA sequencing data, while cell PLMs lack the ability to
  handle free text, restricting their use in multimodal tasks. Existing efforts
  to bridge these modalities often suffer from information loss or inadequate
  single-modal pre-training, leading to suboptimal performances. To address these
  challenges, we propose Single-Cell MultiModal Generative Pre-trained
  Transformer (scMMGPT), a unified PLM for joint cell and text modeling. scMMGPT
  effectively integrates the state-of-the-art cell and text PLMs, facilitating
  cross-modal knowledge sharing for improved performance. To bridge the text-cell
  modality gap, scMMGPT leverages dedicated cross-modal projectors, and undergoes
  extensive pre-training on 27 million cells -- the largest dataset for
  multimodal cell-text PLMs to date. This large-scale pre-training enables
  scMMGPT to excel in joint cell-text tasks, achieving an 84\% relative
  improvement of textual discrepancy for cell description generation, 20.5\%
  higher accuracy for cell type annotation, and 4\% improvement in k-NN
  accuracy for text-conditioned pseudo-cell generation, outperforming baselines.
paper_authors._id: 67d247c2a48964532e40e78e
paper_authors.name: Yaorui Shi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-14T11:26:21.592Z
paper_authors._id: 63edd2d1f765928ceeb49057
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1676530369930-noauth.png
paper_authors.fullname: Yaorui SHI
paper_authors.type: user
paper_authors.user: yrshi
paper_authors._id: 67d247c2a48964532e40e78f
paper_authors.name: Jiaqi Yang
paper_authors._id: 67d247c2a48964532e40e790
paper_authors.name: Sihang Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-14T11:27:04.999Z
paper_authors._id: 635913a68f38318bfabc90ea
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/635913a68f38318bfabc90ea/29M50hPy2nMwMjMElAEgA.jpeg
paper_authors.fullname: Sihang Li
paper_authors.type: user
paper_authors.user: Sihangli
paper_authors._id: 67d247c2a48964532e40e791
paper_authors.name: Junfeng Fang
paper_authors._id: 67d247c2a48964532e40e792
paper_authors.name: Xiang Wang
paper_authors._id: 67d247c2a48964532e40e793
paper_authors.name: Zhiyuan Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-14T11:26:44.031Z
paper_authors._id: 6310a3cd531cc21f9e06de6a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6310a3cd531cc21f9e06de6a/aTGMx3O41lUARK9s3dAik.jpeg
paper_authors.fullname: Zhiyuan Liu
paper_authors.type: user
paper_authors.user: acharkq
paper_authors._id: 67d247c2a48964532e40e794
paper_authors.name: Yang Zhang
paper_title: Multimodal Language Modeling for High-Accuracy Single Cell
    Transcriptomics Analysis and Generation
paper_summary: Pre-trained language models (PLMs) have revolutionized scientific research,
  yet their application to single-cell analysis remains limited. Text PLMs cannot
  process single-cell RNA sequencing data, while cell PLMs lack the ability to
  handle free text, restricting their use in multimodal tasks. Existing efforts
  to bridge these modalities often suffer from information loss or inadequate
  single-modal pre-training, leading to suboptimal performances. To address these
  challenges, we propose Single-Cell MultiModal Generative Pre-trained
  Transformer (scMMGPT), a unified PLM for joint cell and text modeling. scMMGPT
  effectively integrates the state-of-the-art cell and text PLMs, facilitating
  cross-modal knowledge sharing for improved performance. To bridge the text-cell
  modality gap, scMMGPT leverages dedicated cross-modal projectors, and undergoes
  extensive pre-training on 27 million cells -- the largest dataset for
  multimodal cell-text PLMs to date. This large-scale pre-training enables
  scMMGPT to excel in joint cell-text tasks, achieving an 84\% relative
  improvement of textual discrepancy for cell description generation, 20.5\%
  higher accuracy for cell type annotation, and 4\% improvement in k-NN
  accuracy for text-conditioned pseudo-cell generation, outperforming baselines.
paper_ai_keywords: single-cell RNA sequencing data pre-trained language models (PLMs) cell PLMs cross-modal tasks information loss single-modal pre-training Single-Cell MultiModal Generative Pre-trained Transformer (scMMGPT) cross-modal projectors multimodal cell-text PLMs cell description generation cell type annotation $k$-NN accuracy text-conditioned pseudo-cell generation