title: Small Language Model Meets with Reinforced Vision Vocabulary
summary: Playing Large Vision Language Models (LVLMs) in 2023 is trendy among the AI
  community. However, the relatively large number of parameters (more than 7B) of
  popular LVLMs makes it difficult to train and deploy on consumer GPUs,
  discouraging many researchers with limited resources. Imagine how cool it would
  be to experience all the features of current LVLMs on an old GTX1080ti (our
  only game card). Accordingly, we present Vary-toy in this report, a small-size
  Vary along with Qwen-1.8B as the base ``large'' language model. In Vary-toy, we
  introduce an improved vision vocabulary, allowing the model to not only possess
  all features of Vary but also gather more generality. Specifically, we replace
  negative samples of natural images with positive sample data driven by object
  detection in the procedure of generating vision vocabulary, more sufficiently
  utilizing the capacity of the vocabulary network and enabling it to efficiently
  encode visual information corresponding to natural objects. For experiments,
  Vary-toy can achieve 65.6% ANLS on DocVQA, 59.1% accuracy on ChartQA, 88.1%
  accuracy on RefCOCO, and 29% on MMVet. The code will be publicly available on
  the homepage.
paper_authors._id: 65b07b8599d765b854da1e6b
paper_authors.name: Haoran Wei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-24T09:01:17.625Z
paper_authors._id: 6436618aeef1f55654a9f458
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6436618aeef1f55654a9f458/OvxGtuDg2GAFG9As-2hzW.jpeg
paper_authors.fullname: Haoran Wei
paper_authors.type: user
paper_authors.user: HaoranWei
paper_authors._id: 65b07b8599d765b854da1e6c
paper_authors.name: Lingyu Kong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-24T09:01:58.409Z
paper_authors._id: 650d23b69169ea733178f508
paper_authors.avatarUrl: /avatars/5f9a7ab6b6a44e74c3eb6a59da2e21ed.svg
paper_authors.fullname: Lingyu Kong
paper_authors.type: user
paper_authors.user: kppkkp
paper_authors._id: 65b07b8599d765b854da1e6d
paper_authors.name: Jinyue Chen
paper_authors._id: 65b07b8599d765b854da1e6e
paper_authors.name: Liang Zhao
paper_authors._id: 65b07b8599d765b854da1e6f
paper_authors.name: Zheng Ge
paper_authors._id: 65b07b8599d765b854da1e70
paper_authors.name: En Yu
paper_authors._id: 65b07b8599d765b854da1e71
paper_authors.name: Jianjian Sun
paper_authors._id: 65b07b8599d765b854da1e72
paper_authors.name: Chunrui Han
paper_authors._id: 65b07b8599d765b854da1e73
paper_authors.name: Xiangyu Zhang
paper_title: Small Language Model Meets with Reinforced Vision Vocabulary
paper_summary: Playing Large Vision Language Models (LVLMs) in 2023 is trendy among the AI
  community. However, the relatively large number of parameters (more than 7B) of
  popular LVLMs makes it difficult to train and deploy on consumer GPUs,
  discouraging many researchers with limited resources. Imagine how cool it would
  be to experience all the features of current LVLMs on an old GTX1080ti (our
  only game card). Accordingly, we present Vary-toy in this report, a small-size
  Vary along with Qwen-1.8B as the base ``large'' language model. In Vary-toy, we
  introduce an improved vision vocabulary, allowing the model to not only possess
  all features of Vary but also gather more generality. Specifically, we replace
  negative samples of natural images with positive sample data driven by object
  detection in the procedure of generating vision vocabulary, more sufficiently
  utilizing the capacity of the vocabulary network and enabling it to efficiently
  encode visual information corresponding to natural objects. For experiments,
  Vary-toy can achieve 65.6% ANLS on DocVQA, 59.1% accuracy on ChartQA, 88.1%
  accuracy on RefCOCO, and 29% on MMVet. The code will be publicly available on
  the homepage.
paper_ai_keywords: Large Vision Language Models (LVLMs) representation engineering (RepE) Improved vision vocabulary negative samples positive samples object detection ANLS (Answer NLG Success) DocVQA (Document Visual Question Answering) ChartQA RefCOCO (Referring Expressions in Context Corpus) MMVet (Multimodal Visual Explainability Task)