title: Vary: Scaling up the Vision Vocabulary for Large Vision-Language Models
summary: Modern Large Vision-Language Models (LVLMs) enjoy the same vision vocabulary
  -- CLIP, which can cover most common vision tasks. However, for some special
  vision task that needs dense and fine-grained vision perception, e.g.,
  document-level OCR or chart understanding, especially in non-English scenarios,
  the CLIP-style vocabulary may encounter low efficiency in tokenizing the vision
  knowledge and even suffer out-of-vocabulary problem. Accordingly, we propose
  Vary, an efficient and effective method to scale up the vision vocabulary of
  LVLMs. The procedures of Vary are naturally divided into two folds: the
  generation and integration of a new vision vocabulary. In the first phase, we
  devise a vocabulary network along with a tiny decoder-only transformer to
  produce the desired vocabulary via autoregression. In the next, we scale up the
  vanilla vision vocabulary by merging the new one with the original one (CLIP),
  enabling the LVLMs can quickly garner new features. Compared to the popular
  BLIP-2, MiniGPT4, and LLaVA, Vary can maintain its vanilla capabilities while
  enjoying more excellent fine-grained perception and understanding ability.
  Specifically, Vary is competent in new document parsing features (OCR or
  markdown conversion) while achieving 78.2% ANLS in DocVQA and 36.2% in MMVet.
  Our code will be publicly available on the homepage.
paper_authors._id: 6577dcc49fae206bdf9fe91d
paper_authors.name: Haoran Wei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-12T08:09:47.086Z
paper_authors._id: 6436618aeef1f55654a9f458
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6436618aeef1f55654a9f458/OvxGtuDg2GAFG9As-2hzW.jpeg
paper_authors.fullname: Haoran Wei
paper_authors.type: user
paper_authors.user: HaoranWei
paper_authors._id: 6577dcc49fae206bdf9fe91e
paper_authors.name: Lingyu Kong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-18T08:11:22.548Z
paper_authors._id: 650d23b69169ea733178f508
paper_authors.avatarUrl: /avatars/5f9a7ab6b6a44e74c3eb6a59da2e21ed.svg
paper_authors.fullname: Lingyu Kong
paper_authors.type: user
paper_authors.user: kppkkp
paper_authors._id: 6577dcc49fae206bdf9fe91f
paper_authors.name: Jinyue Chen
paper_authors._id: 6577dcc49fae206bdf9fe920
paper_authors.name: Liang Zhao
paper_authors._id: 6577dcc49fae206bdf9fe921
paper_authors.name: Zheng Ge
paper_authors._id: 6577dcc49fae206bdf9fe922
paper_authors.name: Jinrong Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-12T08:11:36.650Z
paper_authors._id: 650bd10c1f2949b99e65ef23
paper_authors.avatarUrl: /avatars/8f36f844690b64bcc7c358063f2bdc3b.svg
paper_authors.fullname: Jinrong Yang
paper_authors.type: user
paper_authors.user: YancieYJR
paper_authors._id: 6577dcc49fae206bdf9fe923
paper_authors.name: Jianjian Sun
paper_authors._id: 6577dcc49fae206bdf9fe924
paper_authors.name: Chunrui Han
paper_authors._id: 6577dcc49fae206bdf9fe925
paper_authors.name: Xiangyu Zhang
paper_title: Vary: Scaling up the Vision Vocabulary for Large Vision-Language Models
paper_summary: Modern Large Vision-Language Models (LVLMs) enjoy the same vision vocabulary
  -- CLIP, which can cover most common vision tasks. However, for some special
  vision task that needs dense and fine-grained vision perception, e.g.,
  document-level OCR or chart understanding, especially in non-English scenarios,
  the CLIP-style vocabulary may encounter low efficiency in tokenizing the vision
  knowledge and even suffer out-of-vocabulary problem. Accordingly, we propose
  Vary, an efficient and effective method to scale up the vision vocabulary of
  LVLMs. The procedures of Vary are naturally divided into two folds: the
  generation and integration of a new vision vocabulary. In the first phase, we
  devise a vocabulary network along with a tiny decoder-only transformer to
  produce the desired vocabulary via autoregression. In the next, we scale up the
  vanilla vision vocabulary by merging the new one with the original one (CLIP),
  enabling the LVLMs can quickly garner new features. Compared to the popular
  BLIP-2, MiniGPT4, and LLaVA, Vary can maintain its vanilla capabilities while
  enjoying more excellent fine-grained perception and understanding ability.
  Specifically, Vary is competent in new document parsing features (OCR or
  markdown conversion) while achieving 78.2% ANLS in DocVQA and 36.2% in MMVet.
  Our code will be publicly available on the homepage.
paper_ai_keywords: Large Vision-Language models (LVLMs) CLIP vision vocabulary vision tasks dense and fine-grained vision perception document-level OCR chart understanding tokenizing Vary noise constraint out-of-vocabulary problem generation of a new vision vocabulary integration of a new vision vocabulary autoregression vanilla vision vocabulary MiniGPT4 LLaVA DocVQA MMVet ANLS ANLED