CoMP: Continual Multimodal Pre-training for Vision Foundation Models
Pre-trained Vision Foundation Models (VFMs) provide strong visual
representations for a wide range of applications. In this paper, we continually
pre-train prevailing VFMs in a multimodal manner such that they can
effortlessly process visual inputs of varying sizes and produce visual
representations that are more aligned with language representations, regardless
of their original pre-training process. To this end, we introduce CoMP, a
carefully designed multimodal pre-training pipeline. CoMP uses a Continual
Rotary Position Embedding to support native resolution continual pre-training,
and an Alignment Loss between visual and textual features through language
prototypes to align multimodal representations. By three-stage training, our
VFMs achieve remarkable improvements not only in multimodal understanding but
also in other downstream tasks such as classification and segmentation.
Remarkably, CoMP-SigLIP achieves scores of 66.7 on ChartQA and 75.9 on DocVQA
with a 0.5B LLM, while maintaining an 87.4% accuracy on ImageNet-1K and a 49.5
mIoU on ADE20K under frozen chunk evaluation.
67e25c4d1908043170bd551d
Yitong Chen
claimed_verified
2025-03-25T08:18:45.692Z
64651db3611ae99d14d392ea
/avatars/b818dc0dddc999758ab5737d5053e8c3.svg
cyt
user
Row11n
67e25c4d1908043170bd551e
Lingchen Meng
67e25c4d1908043170bd551f
Wujian Peng
claimed_verified
2025-03-26T20:49:16.314Z
649bce4f200e2dff194d9883
/avatars/b55a8bdc6f7e2bf9de5f26dc1d87bee3.svg
Wujian Peng
user
wjpoom
67e25c4d1908043170bd5520
Zuxuan Wu
67e25c4d1908043170bd5521
Yu-Gang Jiang
CoMP: Continual Multimodal Pre-training for Vision Foundation Models
Pre-trained Vision Foundation Models (VFMs) provide strong visual
representations for a wide range of applications. In this paper, we continually
pre-train prevailing VFMs in a multimodal manner such that they can
effortlessly process visual inputs of varying sizes and produce visual
representations that are more aligned with language representations, regardless
of their original pre-training process. To this end, we introduce CoMP, a
carefully designed multimodal pre-training pipeline. CoMP uses a Continual
Rotary Position Embedding to support native resolution continual pre-training,
and an Alignment Loss between visual and textual features through language
prototypes to align multimodal representations. By three-stage training, our
VFMs achieve remarkable improvements not only in multimodal understanding but
also in other downstream tasks such as classification and segmentation.
Remarkably, CoMP-SigLIP achieves scores of 66.7 on ChartQA and 75.9 on DocVQA
with a 0.5B LLM, while maintaining an 87.4% accuracy on ImageNet-1K and a 49.5
mIoU on ADE20K under frozen chunk evaluation.
CoMP, a multimodal pre-training pipeline with Continual Rotary Position Embedding and Alignment Loss, enhances visual representations for various tasks by aligning with language models.
Pre-trained Vision Foundation Models Continual Rotary Position Embedding Multimodal pre-training pipeline Alignment Loss language prototypes multimodal understanding ChartQA DocVQA ImageNet-1K ADE20K frozen chunk evaluation