Small Language Model Meets with Reinforced Vision Vocabulary
Playing Large Vision Language Models (LVLMs) in 2023 is trendy among the AI
community. However, the relatively large number of parameters (more than 7B) of
popular LVLMs makes it difficult to train and deploy on consumer GPUs,
discouraging many researchers with limited resources. Imagine how cool it would
be to experience all the features of current LVLMs on an old GTX1080ti (our
only game card). Accordingly, we present Vary-toy in this report, a small-size
Vary along with Qwen-1.8B as the base ``large'' language model. In Vary-toy, we
introduce an improved vision vocabulary, allowing the model to not only possess
all features of Vary but also gather more generality. Specifically, we replace
negative samples of natural images with positive sample data driven by object
detection in the procedure of generating vision vocabulary, more sufficiently
utilizing the capacity of the vocabulary network and enabling it to efficiently
encode visual information corresponding to natural objects. For experiments,
Vary-toy can achieve 65.6% ANLS on DocVQA, 59.1% accuracy on ChartQA, 88.1%
accuracy on RefCOCO, and 29% on MMVet. The code will be publicly available on
the homepage.
65b07b8599d765b854da1e6b
Haoran Wei
admin_assigned
2024-01-24T09:01:17.625Z
6436618aeef1f55654a9f458
https://cdn-avatars.huggingface.co/v1/production/uploads/6436618aeef1f55654a9f458/OvxGtuDg2GAFG9As-2hzW.jpeg
Haoran Wei
user
HaoranWei
65b07b8599d765b854da1e6c
Lingyu Kong
admin_assigned
2024-01-24T09:01:58.409Z
650d23b69169ea733178f508
/avatars/5f9a7ab6b6a44e74c3eb6a59da2e21ed.svg
Lingyu Kong
user
kppkkp
65b07b8599d765b854da1e6d
Jinyue Chen
65b07b8599d765b854da1e6e
Liang Zhao
65b07b8599d765b854da1e6f
Zheng Ge
65b07b8599d765b854da1e70
En Yu
65b07b8599d765b854da1e71
Jianjian Sun
65b07b8599d765b854da1e72
Chunrui Han
65b07b8599d765b854da1e73
Xiangyu Zhang
Small Language Model Meets with Reinforced Vision Vocabulary
Playing Large Vision Language Models (LVLMs) in 2023 is trendy among the AI
community. However, the relatively large number of parameters (more than 7B) of
popular LVLMs makes it difficult to train and deploy on consumer GPUs,
discouraging many researchers with limited resources. Imagine how cool it would
be to experience all the features of current LVLMs on an old GTX1080ti (our
only game card). Accordingly, we present Vary-toy in this report, a small-size
Vary along with Qwen-1.8B as the base ``large'' language model. In Vary-toy, we
introduce an improved vision vocabulary, allowing the model to not only possess
all features of Vary but also gather more generality. Specifically, we replace
negative samples of natural images with positive sample data driven by object
detection in the procedure of generating vision vocabulary, more sufficiently
utilizing the capacity of the vocabulary network and enabling it to efficiently
encode visual information corresponding to natural objects. For experiments,
Vary-toy can achieve 65.6% ANLS on DocVQA, 59.1% accuracy on ChartQA, 88.1%
accuracy on RefCOCO, and 29% on MMVet. The code will be publicly available on
the homepage.
Vary-toy, a smaller version of Vary using Qwen-1.8B, enhances vision vocabulary through object detection and achieves competitive performance on visual question answering tasks.
Vary Qwen-1.8B vision vocabulary object detection ANLS DocVQA ChartQA RefCOCO MMVet