title: Open-Qwen2VL: Compute-Efficient Pre-Training of Fully-Open Multimodal
    LLMs on Academic Resources
summary: The reproduction of state-of-the-art multimodal LLM pre-training faces
  barriers at every stage of the pipeline, including high-quality data filtering,
  multimodal data mixture strategies, sequence packing techniques, and training
  frameworks. We introduce Open-Qwen2VL, a fully open-source 2B-parameter
  Multimodal Large Language Model pre-trained efficiently on 29M image-text pairs
  using only 442 A100-40G GPU hours. Our approach employs low-to-high dynamic
  image resolution and multimodal sequence packing to significantly enhance
  pre-training efficiency. The training dataset was carefully curated using both
  MLLM-based filtering techniques (e.g., MLM-Filter) and conventional CLIP-based
  filtering methods, substantially improving data quality and training
  efficiency. The Open-Qwen2VL pre-training is conducted on academic level
  8xA100-40G GPUs at UCSB on 5B packed multimodal tokens, which is 0.36\% of 1.4T
  multimodal pre-training tokens of Qwen2-VL. The final instruction-tuned
  Open-Qwen2VL outperforms partially-open state-of-the-art MLLM Qwen2-VL-2B on
  various multimodal benchmarks of MMBench, SEEDBench, MMstar, and MathVista,
  indicating the remarkable training efficiency of Open-Qwen2VL. We open-source
  all aspects of our work, including compute-efficient and data-efficient
  training details, data filtering methods, sequence packing scripts,
  pre-training data in WebDataset format, FSDP-based training codebase, and both
  base and instruction-tuned model checkpoints. We redefine "fully open" for
  multimodal LLMs as the complete release of: 1) the training codebase, 2)
  detailed data filtering techniques, and 3) all pre-training and supervised
  fine-tuning data used to develop the model.
paper_authors._id: 67ecaf516560da48c5c34106
paper_authors.name: Weizhi Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-02T08:22:20.594Z
paper_authors._id: 63d34004b734eaa4d4faeccf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63d34004b734eaa4d4faeccf/aH7F-xn4PMQjHHcvL-0vs.jpeg
paper_authors.fullname: Weizhi Wang
paper_authors.type: user
paper_authors.user: weizhiwang
paper_authors._id: 67ecaf516560da48c5c34107
paper_authors.name: Yu Tian
paper_authors._id: 67ecaf516560da48c5c34108
paper_authors.name: Linjie Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-02T09:55:04.382Z
paper_authors._id: 67cd0b291580ba5d5ee65ffd
paper_authors.avatarUrl: /avatars/9584a55473868e5ca1fa09b1536ca546.svg
paper_authors.fullname: yanglinjie
paper_authors.type: user
paper_authors.user: yanglj55
paper_authors._id: 67ecaf516560da48c5c34109
paper_authors.name: Heng Wang
paper_authors._id: 67ecaf516560da48c5c3410a
paper_authors.name: Xifeng Yan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-02T09:54:38.935Z
paper_authors._id: 65cd4785c40ab294321d610e
paper_authors.avatarUrl: /avatars/3f0053aa2b3d90a10b60ab24cf575fd5.svg
paper_authors.fullname: Xifeng Yan
paper_authors.type: user
paper_authors.user: windwest
paper_title: Open-Qwen2VL: Compute-Efficient Pre-Training of Fully-Open Multimodal
    LLMs on Academic Resources
paper_summary: The reproduction of state-of-the-art multimodal LLM pre-training faces
  barriers at every stage of the pipeline, including high-quality data filtering,
  multimodal data mixture strategies, sequence packing techniques, and training
  frameworks. We introduce Open-Qwen2VL, a fully open-source 2B-parameter
  Multimodal Large Language Model pre-trained efficiently on 29M image-text pairs
  using only 442 A100-40G GPU hours. Our approach employs low-to-high dynamic
  image resolution and multimodal sequence packing to significantly enhance
  pre-training efficiency. The training dataset was carefully curated using both
  MLLM-based filtering techniques (e.g., MLM-Filter) and conventional CLIP-based
  filtering methods, substantially improving data quality and training
  efficiency. The Open-Qwen2VL pre-training is conducted on academic level
  8xA100-40G GPUs at UCSB on 5B packed multimodal tokens, which is 0.36\% of 1.4T
  multimodal pre-training tokens of Qwen2-VL. The final instruction-tuned
  Open-Qwen2VL outperforms partially-open state-of-the-art MLLM Qwen2-VL-2B on
  various multimodal benchmarks of MMBench, SEEDBench, MMstar, and MathVista,
  indicating the remarkable training efficiency of Open-Qwen2VL. We open-source
  all aspects of our work, including compute-efficient and data-efficient
  training details, data filtering methods, sequence packing scripts,
  pre-training data in WebDataset format, FSDP-based training codebase, and both
  base and instruction-tuned model checkpoints. We redefine "fully open" for
  multimodal LLMs as the complete release of: 1) the training codebase, 2)
  detailed data filtering techniques, and 3) all pre-training and supervised
  fine-tuning data used to develop the model.
paper_ai_summary: Open-Qwen2VL is a 2B-parameter multimodal LLM pre-trained efficiently on image-text pairs with improved data filtering and sequence packing, outperforming state-of-the-art models on various benchmarks.
paper_ai_keywords: multimodal LLM pre-training image resolution multimodal sequence packing MLM-based filtering CLIP-based filtering academic level 8xA100-40G GPUs FSDP-based training WebDataset MMBench SEEDBench MMstar MathVista compute-efficient data-efficient