title: From Head to Tail: Towards Balanced Representation in Large
    Vision-Language Models through Adaptive Data Calibration
summary: Large Vision-Language Models (LVLMs) have achieved significant progress in
  combining visual comprehension with language generation. Despite this success,
  the training data of LVLMs still suffers from Long-Tail (LT) problems, where
  the data distribution is highly imbalanced. Previous works have mainly focused
  on traditional VLM architectures, i.e., CLIP or ViT, and specific tasks such as
  recognition and classification. Nevertheless, the exploration of LVLM (e.g.
  LLaVA) and more general tasks (e.g. Visual Question Answering and Visual
  Reasoning) remains under-explored. In this paper, we first conduct an in-depth
  analysis of the LT issues in LVLMs and identify two core causes: the
  overrepresentation of head concepts and the underrepresentation of tail
  concepts. Based on the above observation, we propose an Adaptive
  Data Refinement Framework (ADR), which
  consists of two stages: Data Rebalancing (DR)
  and Data Synthesis (DS). In the DR stage, we
  adaptively rebalance the redundant data based on entity distributions, while in
  the DS stage, we leverage Denoising Diffusion Probabilistic Models (DDPMs) and
  scarce images to supplement underrepresented portions. Through comprehensive
  evaluations across eleven benchmarks, our proposed ADR effectively mitigates
  the long-tail problem in the training data, improving the average performance
  of LLaVA 1.5 relatively by 4.36%, without increasing the training data volume.
paper_authors._id: 67e0cc432bbf376bdb18623b
paper_authors.name: Mingyang Song
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-24T08:07:01.991Z
paper_authors._id: 66aca01e33f6b27979856f6f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/66aca01e33f6b27979856f6f/IyOxv89TudwscGH7tdue3.jpeg
paper_authors.fullname: Mingyang Song
paper_authors.type: user
paper_authors.user: hitsmy
paper_authors._id: 67e0cc432bbf376bdb18623c
paper_authors.name: Xiaoye Qu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-24T12:51:10.424Z
paper_authors._id: 64cb54da1af278541d663708
paper_authors.avatarUrl: /avatars/c44507cc92bb2e83154bad31b90ce6dd.svg
paper_authors.fullname: Xiaoye Qu
paper_authors.type: user
paper_authors.user: Xiaoye08
paper_authors._id: 67e0cc432bbf376bdb18623d
paper_authors.name: Jiawei Zhou
paper_authors._id: 67e0cc432bbf376bdb18623e
paper_authors.name: Yu Cheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-02T07:52:00.997Z
paper_authors._id: 67017abfe4d49b157ac534d9
paper_authors.avatarUrl: /avatars/997e1b9f54b27a7728a9d4abfee4ba91.svg
paper_authors.fullname: Yu Cheng
paper_authors.type: user
paper_authors.user: ych133
paper_title: From Head to Tail: Towards Balanced Representation in Large
    Vision-Language Models through Adaptive Data Calibration
paper_summary: Large Vision-Language Models (LVLMs) have achieved significant progress in
  combining visual comprehension with language generation. Despite this success,
  the training data of LVLMs still suffers from Long-Tail (LT) problems, where
  the data distribution is highly imbalanced. Previous works have mainly focused
  on traditional VLM architectures, i.e., CLIP or ViT, and specific tasks such as
  recognition and classification. Nevertheless, the exploration of LVLM (e.g.
  LLaVA) and more general tasks (e.g. Visual Question Answering and Visual
  Reasoning) remains under-explored. In this paper, we first conduct an in-depth
  analysis of the LT issues in LVLMs and identify two core causes: the
  overrepresentation of head concepts and the underrepresentation of tail
  concepts. Based on the above observation, we propose an Adaptive
  Data Refinement Framework (ADR), which
  consists of two stages: Data Rebalancing (DR)
  and Data Synthesis (DS). In the DR stage, we
  adaptively rebalance the redundant data based on entity distributions, while in
  the DS stage, we leverage Denoising Diffusion Probabilistic Models (DDPMs) and
  scarce images to supplement underrepresented portions. Through comprehensive
  evaluations across eleven benchmarks, our proposed ADR effectively mitigates
  the long-tail problem in the training data, improving the average performance
  of LLaVA 1.5 relatively by 4.36%, without increasing the training data volume.
paper_ai_summary: The proposed ADR framework improves performance in large vision-language models by addressing long-tail data distribution issues through data rebalancing and synthesis using denoising diffusion probabilistic models.
paper_ai_keywords: Large Vision-Language Models (LVLMs) Long-Tail (LT) problems VLM architectures CLIP ViT LLaVA Visual Question Answering Visual Reasoning Adaptive Data Refinement Framework (ADR) Data Rebalancing (DR) Data Synthesis (DS) Denoising Diffusion Probabilistic Models (DDPMs)