title: CoMP: Continual Multimodal Pre-training for Vision Foundation Models
summary: Pre-trained Vision Foundation Models (VFMs) provide strong visual
  representations for a wide range of applications. In this paper, we continually
  pre-train prevailing VFMs in a multimodal manner such that they can
  effortlessly process visual inputs of varying sizes and produce visual
  representations that are more aligned with language representations, regardless
  of their original pre-training process. To this end, we introduce CoMP, a
  carefully designed multimodal pre-training pipeline. CoMP uses a Continual
  Rotary Position Embedding to support native resolution continual pre-training,
  and an Alignment Loss between visual and textual features through language
  prototypes to align multimodal representations. By three-stage training, our
  VFMs achieve remarkable improvements not only in multimodal understanding but
  also in other downstream tasks such as classification and segmentation.
  Remarkably, CoMP-SigLIP achieves scores of 66.7 on ChartQA and 75.9 on DocVQA
  with a 0.5B LLM, while maintaining an 87.4% accuracy on ImageNet-1K and a 49.5
  mIoU on ADE20K under frozen chunk evaluation.
paper_authors._id: 67e25c4d1908043170bd551d
paper_authors.name: Yitong Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-25T08:18:45.692Z
paper_authors._id: 64651db3611ae99d14d392ea
paper_authors.avatarUrl: /avatars/b818dc0dddc999758ab5737d5053e8c3.svg
paper_authors.fullname: cyt
paper_authors.type: user
paper_authors.user: Row11n
paper_authors._id: 67e25c4d1908043170bd551e
paper_authors.name: Lingchen Meng
paper_authors._id: 67e25c4d1908043170bd551f
paper_authors.name: Wujian Peng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-26T20:49:16.314Z
paper_authors._id: 649bce4f200e2dff194d9883
paper_authors.avatarUrl: /avatars/b55a8bdc6f7e2bf9de5f26dc1d87bee3.svg
paper_authors.fullname: Wujian Peng
paper_authors.type: user
paper_authors.user: wjpoom
paper_authors._id: 67e25c4d1908043170bd5520
paper_authors.name: Zuxuan Wu
paper_authors._id: 67e25c4d1908043170bd5521
paper_authors.name: Yu-Gang Jiang
paper_title: CoMP: Continual Multimodal Pre-training for Vision Foundation Models
paper_summary: Pre-trained Vision Foundation Models (VFMs) provide strong visual
  representations for a wide range of applications. In this paper, we continually
  pre-train prevailing VFMs in a multimodal manner such that they can
  effortlessly process visual inputs of varying sizes and produce visual
  representations that are more aligned with language representations, regardless
  of their original pre-training process. To this end, we introduce CoMP, a
  carefully designed multimodal pre-training pipeline. CoMP uses a Continual
  Rotary Position Embedding to support native resolution continual pre-training,
  and an Alignment Loss between visual and textual features through language
  prototypes to align multimodal representations. By three-stage training, our
  VFMs achieve remarkable improvements not only in multimodal understanding but
  also in other downstream tasks such as classification and segmentation.
  Remarkably, CoMP-SigLIP achieves scores of 66.7 on ChartQA and 75.9 on DocVQA
  with a 0.5B LLM, while maintaining an 87.4% accuracy on ImageNet-1K and a 49.5
  mIoU on ADE20K under frozen chunk evaluation.
paper_ai_summary: CoMP, a multimodal pre-training pipeline with Continual Rotary Position Embedding and Alignment Loss, enhances visual representations for various tasks by aligning with language models.
paper_ai_keywords: Pre-trained Vision Foundation Models Continual Rotary Position Embedding Multimodal pre-training pipeline Alignment Loss language prototypes multimodal understanding ChartQA DocVQA ImageNet-1K ADE20K frozen chunk evaluation