COMPACT: COMPositional Atomic-to-Complex Visual Capability Tuning
Multimodal Large Language Models (MLLMs) excel at simple vision-language
tasks but struggle when faced with complex tasks that require multiple
capabilities, such as simultaneously recognizing objects, counting them, and
understanding their spatial relationships. This might be partially the result
of the fact that Visual Instruction Tuning (VIT), a critical training step for
MLLMs, has traditionally focused on scaling data volume, but not the
compositional complexity of training examples. We propose COMPACT
(COMPositional Atomic-to-complex visual Capability Tuning), which generates a
training dataset explicitly controlling for the compositional complexity of the
training examples. The data from COMPACT allows MLLMs to train on combinations
of atomic capabilities to learn complex capabilities more efficiently. Across
all benchmarks, COMPACT achieves comparable performance to the LLaVA-665k VIT
while using less than 10% of its data budget, and even outperforms it on
several, especially those involving complex multi-capability tasks. For
example, COMPACT achieves substantial 83.3% improvement on MMStar and 94.0%
improvement on MM-Vet compared to the full-scale VIT on particularly complex
questions that require four or more atomic capabilities. COMPACT offers a
scalable, data-efficient, visual compositional tuning recipe to improve on
complex visual-language tasks.
6812d4d6ce88881cb51b3b70
Xindi Wu
claimed_verified
2025-05-02T06:35:07.812Z
613940c0905b1938233881e3
https://cdn-avatars.huggingface.co/v1/production/uploads/613940c0905b1938233881e3/Vb4kFWKEq6AILUP9KmCZA.png
Xindi Wu
user
xindiw
6812d4d6ce88881cb51b3b71
Hee Seung Hwang
6812d4d6ce88881cb51b3b72
Polina Kirichenko
6812d4d6ce88881cb51b3b73
Olga Russakovsky
COMPACT: COMPositional Atomic-to-Complex Visual Capability Tuning
Multimodal Large Language Models (MLLMs) excel at simple vision-language
tasks but struggle when faced with complex tasks that require multiple
capabilities, such as simultaneously recognizing objects, counting them, and
understanding their spatial relationships. This might be partially the result
of the fact that Visual Instruction Tuning (VIT), a critical training step for
MLLMs, has traditionally focused on scaling data volume, but not the
compositional complexity of training examples. We propose COMPACT
(COMPositional Atomic-to-complex visual Capability Tuning), which generates a
training dataset explicitly controlling for the compositional complexity of the
training examples. The data from COMPACT allows MLLMs to train on combinations
of atomic capabilities to learn complex capabilities more efficiently. Across
all benchmarks, COMPACT achieves comparable performance to the LLaVA-665k VIT
while using less than 10% of its data budget, and even outperforms it on
several, especially those involving complex multi-capability tasks. For
example, COMPACT achieves substantial 83.3% improvement on MMStar and 94.0%
improvement on MM-Vet compared to the full-scale VIT on particularly complex
questions that require four or more atomic capabilities. COMPACT offers a
scalable, data-efficient, visual compositional tuning recipe to improve on
complex visual-language tasks.
COMPACT, a compositional visual capability tuning method, improves multimodal large language models' performance on complex vision-language tasks with less data than traditional visual instruction tuning.
multimodal large language models MLLMs simple vision-language tasks complex tasks object recognition counting spatial relationships Visual Instruction Tuning VIT compositional complexity atomic capabilities complex capabilities benchmarks LLaVA-665k MMStar MM-Vet