title: POINTS: Improving Your Vision-language Model with Affordable Strategies
summary: In recent years, vision-language models have made significant strides,
  excelling in tasks like optical character recognition and geometric
  problem-solving. However, several critical issues remain: 1) Proprietary models
  often lack transparency about their architectures, while open-source models
  need more detailed ablations of their training strategies. 2) Pre-training data
  in open-source works is under-explored, with datasets added empirically, making
  the process cumbersome. 3) Fine-tuning often focuses on adding datasets,
  leading to diminishing returns. To address these issues, we propose the
  following contributions: 1) We trained a robust baseline model using the latest
  advancements in vision-language models, introducing effective improvements and
  conducting comprehensive ablation and validation for each technique. 2)
  Inspired by recent work on large language models, we filtered pre-training data
  using perplexity, selecting the lowest perplexity data for training. This
  approach allowed us to train on a curated 1M dataset, achieving competitive
  performance. 3) During visual instruction tuning, we used model soup on
  different datasets when adding more datasets yielded marginal improvements.
  These innovations resulted in a 9B parameter model that performs competitively
  with state-of-the-art models. Our strategies are efficient and lightweight,
  making them easily adoptable by the community.
paper_authors._id: 66e016cc7f21553cc014a04d
paper_authors.name: Yuan Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-10T09:56:45.696Z
paper_authors._id: 64d47a7a508a6313e33faedd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64d47a7a508a6313e33faedd/8DrbXdxA-sKZb4jgNhvY1.jpeg
paper_authors.fullname: Yuan Liu
paper_authors.type: user
paper_authors.user: YuanLiuuuuuu
paper_authors._id: 66e016cc7f21553cc014a04e
paper_authors.name: Zhongyin Zhao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-10T09:56:51.234Z
paper_authors._id: 664c11bc533c5d0c9c717828
paper_authors.avatarUrl: /avatars/a2844359c341f41bedfa257b14be5e23.svg
paper_authors.fullname: Zhongyin Zhao
paper_authors.type: user
paper_authors.user: scyr
paper_authors._id: 66e016cc7f21553cc014a04f
paper_authors.name: Ziyuan Zhuang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-10T09:56:57.143Z
paper_authors._id: 6628ff54f9a3545305f0a7d4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/LFPfFE89Pety7eaZ2FWNx.jpeg
paper_authors.fullname: Ziyuan Zhuang
paper_authors.type: user
paper_authors.user: Zeon-Zhuang
paper_authors._id: 66e016cc7f21553cc014a050
paper_authors.name: Le Tian
paper_authors._id: 66e016cc7f21553cc014a051
paper_authors.name: Xiao Zhou
paper_authors._id: 66e016cc7f21553cc014a052
paper_authors.name: Jie Zhou
paper_title: POINTS: Improving Your Vision-language Model with Affordable Strategies
paper_summary: In recent years, vision-language models have made significant strides,
  excelling in tasks like optical character recognition and geometric
  problem-solving. However, several critical issues remain: 1) Proprietary models
  often lack transparency about their architectures, while open-source models
  need more detailed ablations of their training strategies. 2) Pre-training data
  in open-source works is under-explored, with datasets added empirically, making
  the process cumbersome. 3) Fine-tuning often focuses on adding datasets,
  leading to diminishing returns. To address these issues, we propose the
  following contributions: 1) We trained a robust baseline model using the latest
  advancements in vision-language models, introducing effective improvements and
  conducting comprehensive ablation and validation for each technique. 2)
  Inspired by recent work on large language models, we filtered pre-training data
  using perplexity, selecting the lowest perplexity data for training. This
  approach allowed us to train on a curated 1M dataset, achieving competitive
  performance. 3) During visual instruction tuning, we used model soup on
  different datasets when adding more datasets yielded marginal improvements.
  These innovations resulted in a 9B parameter model that performs competitively
  with state-of-the-art models. Our strategies are efficient and lightweight,
  making them easily adoptable by the community.
paper_ai_keywords: vision-language models optical character recognition geometric problem-solving proprietary models cognitive architectures open source models attentions training strategies pre-training data datasets recently-tuning baseline model effective improvements comprehensive ablation validations perplexity model soup parameter-efficient fine-tuning