POINTS: Improving Your Vision-language Model with Affordable Strategies
In recent years, vision-language models have made significant strides,
excelling in tasks like optical character recognition and geometric
problem-solving. However, several critical issues remain: 1) Proprietary models
often lack transparency about their architectures, while open-source models
need more detailed ablations of their training strategies. 2) Pre-training data
in open-source works is under-explored, with datasets added empirically, making
the process cumbersome. 3) Fine-tuning often focuses on adding datasets,
leading to diminishing returns. To address these issues, we propose the
following contributions: 1) We trained a robust baseline model using the latest
advancements in vision-language models, introducing effective improvements and
conducting comprehensive ablation and validation for each technique. 2)
Inspired by recent work on large language models, we filtered pre-training data
using perplexity, selecting the lowest perplexity data for training. This
approach allowed us to train on a curated 1M dataset, achieving competitive
performance. 3) During visual instruction tuning, we used model soup on
different datasets when adding more datasets yielded marginal improvements.
These innovations resulted in a 9B parameter model that performs competitively
with state-of-the-art models. Our strategies are efficient and lightweight,
making them easily adoptable by the community.
66e016cc7f21553cc014a04d
Yuan Liu
admin_assigned
2024-09-10T09:56:45.696Z
64d47a7a508a6313e33faedd
https://cdn-avatars.huggingface.co/v1/production/uploads/64d47a7a508a6313e33faedd/8DrbXdxA-sKZb4jgNhvY1.jpeg
Yuan Liu
user
YuanLiuuuuuu
66e016cc7f21553cc014a04e
Zhongyin Zhao
admin_assigned
2024-09-10T09:56:51.234Z
664c11bc533c5d0c9c717828
/avatars/a2844359c341f41bedfa257b14be5e23.svg
Zhongyin Zhao
user
scyr
66e016cc7f21553cc014a04f
Ziyuan Zhuang
admin_assigned
2024-09-10T09:56:57.143Z
6628ff54f9a3545305f0a7d4
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/LFPfFE89Pety7eaZ2FWNx.jpeg
Ziyuan Zhuang
user
Zeon-Zhuang
66e016cc7f21553cc014a050
Le Tian
66e016cc7f21553cc014a051
Xiao Zhou
66e016cc7f21553cc014a052
Jie Zhou
POINTS: Improving Your Vision-language Model with Affordable Strategies
In recent years, vision-language models have made significant strides,
excelling in tasks like optical character recognition and geometric
problem-solving. However, several critical issues remain: 1) Proprietary models
often lack transparency about their architectures, while open-source models
need more detailed ablations of their training strategies. 2) Pre-training data
in open-source works is under-explored, with datasets added empirically, making
the process cumbersome. 3) Fine-tuning often focuses on adding datasets,
leading to diminishing returns. To address these issues, we propose the
following contributions: 1) We trained a robust baseline model using the latest
advancements in vision-language models, introducing effective improvements and
conducting comprehensive ablation and validation for each technique. 2)
Inspired by recent work on large language models, we filtered pre-training data
using perplexity, selecting the lowest perplexity data for training. This
approach allowed us to train on a curated 1M dataset, achieving competitive
performance. 3) During visual instruction tuning, we used model soup on
different datasets when adding more datasets yielded marginal improvements.
These innovations resulted in a 9B parameter model that performs competitively
with state-of-the-art models. Our strategies are efficient and lightweight,
making them easily adoptable by the community.
We developed a robust vision-language model with a filtered dataset and model soup tuning, achieving competitive performance with a 9B parameter model.
vision-language models optical character recognition geometric problem-solving model soup perplexity ablation validation