From Head to Tail: Towards Balanced Representation in Large
  Vision-Language Models through Adaptive Data Calibration
Large Vision-Language Models (LVLMs) have achieved significant progress in
combining visual comprehension with language generation. Despite this success,
the training data of LVLMs still suffers from Long-Tail (LT) problems, where
the data distribution is highly imbalanced. Previous works have mainly focused
on traditional VLM architectures, i.e., CLIP or ViT, and specific tasks such as
recognition and classification. Nevertheless, the exploration of LVLM (e.g.
LLaVA) and more general tasks (e.g. Visual Question Answering and Visual
Reasoning) remains under-explored. In this paper, we first conduct an in-depth
analysis of the LT issues in LVLMs and identify two core causes: the
overrepresentation of head concepts and the underrepresentation of tail
concepts. Based on the above observation, we propose an Adaptive
Data Refinement Framework (ADR), which
consists of two stages: Data Rebalancing (DR)
and Data Synthesis (DS). In the DR stage, we
adaptively rebalance the redundant data based on entity distributions, while in
the DS stage, we leverage Denoising Diffusion Probabilistic Models (DDPMs) and
scarce images to supplement underrepresented portions. Through comprehensive
evaluations across eleven benchmarks, our proposed ADR effectively mitigates
the long-tail problem in the training data, improving the average performance
of LLaVA 1.5 relatively by 4.36%, without increasing the training data volume.
67e0cc432bbf376bdb18623b
Mingyang Song
claimed_verified
2025-03-24T08:07:01.991Z
66aca01e33f6b27979856f6f
https://cdn-avatars.huggingface.co/v1/production/uploads/66aca01e33f6b27979856f6f/IyOxv89TudwscGH7tdue3.jpeg
Mingyang Song
user
hitsmy
67e0cc432bbf376bdb18623c
Xiaoye Qu
admin_assigned
2025-03-24T12:51:10.424Z
64cb54da1af278541d663708
/avatars/c44507cc92bb2e83154bad31b90ce6dd.svg
Xiaoye Qu
user
Xiaoye08
67e0cc432bbf376bdb18623d
Jiawei Zhou
67e0cc432bbf376bdb18623e
Yu Cheng
claimed_verified
2025-06-02T07:52:00.997Z
67017abfe4d49b157ac534d9
/avatars/997e1b9f54b27a7728a9d4abfee4ba91.svg
Yu Cheng
user
ych133
From Head to Tail: Towards Balanced Representation in Large
  Vision-Language Models through Adaptive Data Calibration
Large Vision-Language Models (LVLMs) have achieved significant progress in
combining visual comprehension with language generation. Despite this success,
the training data of LVLMs still suffers from Long-Tail (LT) problems, where
the data distribution is highly imbalanced. Previous works have mainly focused
on traditional VLM architectures, i.e., CLIP or ViT, and specific tasks such as
recognition and classification. Nevertheless, the exploration of LVLM (e.g.
LLaVA) and more general tasks (e.g. Visual Question Answering and Visual
Reasoning) remains under-explored. In this paper, we first conduct an in-depth
analysis of the LT issues in LVLMs and identify two core causes: the
overrepresentation of head concepts and the underrepresentation of tail
concepts. Based on the above observation, we propose an Adaptive
Data Refinement Framework (ADR), which
consists of two stages: Data Rebalancing (DR)
and Data Synthesis (DS). In the DR stage, we
adaptively rebalance the redundant data based on entity distributions, while in
the DS stage, we leverage Denoising Diffusion Probabilistic Models (DDPMs) and
scarce images to supplement underrepresented portions. Through comprehensive
evaluations across eleven benchmarks, our proposed ADR effectively mitigates
the long-tail problem in the training data, improving the average performance
of LLaVA 1.5 relatively by 4.36%, without increasing the training data volume.
The proposed ADR framework improves performance in large vision-language models by addressing long-tail data distribution issues through data rebalancing and synthesis using denoising diffusion probabilistic models.
Large Vision-Language Models (LVLMs) Long-Tail (LT) problems VLM architectures CLIP ViT LLaVA Visual Question Answering Visual Reasoning Adaptive Data Refinement Framework (ADR) Data Rebalancing (DR) Data Synthesis (DS) Denoising Diffusion Probabilistic Models (DDPMs)