title: COMPACT: COMPositional Atomic-to-Complex Visual Capability Tuning
summary: Multimodal Large Language Models (MLLMs) excel at simple vision-language
  tasks but struggle when faced with complex tasks that require multiple
  capabilities, such as simultaneously recognizing objects, counting them, and
  understanding their spatial relationships. This might be partially the result
  of the fact that Visual Instruction Tuning (VIT), a critical training step for
  MLLMs, has traditionally focused on scaling data volume, but not the
  compositional complexity of training examples. We propose COMPACT
  (COMPositional Atomic-to-complex visual Capability Tuning), which generates a
  training dataset explicitly controlling for the compositional complexity of the
  training examples. The data from COMPACT allows MLLMs to train on combinations
  of atomic capabilities to learn complex capabilities more efficiently. Across
  all benchmarks, COMPACT achieves comparable performance to the LLaVA-665k VIT
  while using less than 10% of its data budget, and even outperforms it on
  several, especially those involving complex multi-capability tasks. For
  example, COMPACT achieves substantial 83.3% improvement on MMStar and 94.0%
  improvement on MM-Vet compared to the full-scale VIT on particularly complex
  questions that require four or more atomic capabilities. COMPACT offers a
  scalable, data-efficient, visual compositional tuning recipe to improve on
  complex visual-language tasks.
paper_authors._id: 6812d4d6ce88881cb51b3b70
paper_authors.name: Xindi Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-02T06:35:07.812Z
paper_authors._id: 613940c0905b1938233881e3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/613940c0905b1938233881e3/Vb4kFWKEq6AILUP9KmCZA.png
paper_authors.fullname: Xindi Wu
paper_authors.type: user
paper_authors.user: xindiw
paper_authors._id: 6812d4d6ce88881cb51b3b71
paper_authors.name: Hee Seung Hwang
paper_authors._id: 6812d4d6ce88881cb51b3b72
paper_authors.name: Polina Kirichenko
paper_authors._id: 6812d4d6ce88881cb51b3b73
paper_authors.name: Olga Russakovsky
paper_title: COMPACT: COMPositional Atomic-to-Complex Visual Capability Tuning
paper_summary: Multimodal Large Language Models (MLLMs) excel at simple vision-language
  tasks but struggle when faced with complex tasks that require multiple
  capabilities, such as simultaneously recognizing objects, counting them, and
  understanding their spatial relationships. This might be partially the result
  of the fact that Visual Instruction Tuning (VIT), a critical training step for
  MLLMs, has traditionally focused on scaling data volume, but not the
  compositional complexity of training examples. We propose COMPACT
  (COMPositional Atomic-to-complex visual Capability Tuning), which generates a
  training dataset explicitly controlling for the compositional complexity of the
  training examples. The data from COMPACT allows MLLMs to train on combinations
  of atomic capabilities to learn complex capabilities more efficiently. Across
  all benchmarks, COMPACT achieves comparable performance to the LLaVA-665k VIT
  while using less than 10% of its data budget, and even outperforms it on
  several, especially those involving complex multi-capability tasks. For
  example, COMPACT achieves substantial 83.3% improvement on MMStar and 94.0%
  improvement on MM-Vet compared to the full-scale VIT on particularly complex
  questions that require four or more atomic capabilities. COMPACT offers a
  scalable, data-efficient, visual compositional tuning recipe to improve on
  complex visual-language tasks.
paper_ai_summary: COMPACT, a compositional visual capability tuning method, improves multimodal large language models' performance on complex vision-language tasks with less data than traditional visual instruction tuning.
paper_ai_keywords: multimodal large language models MLLMs simple vision-language tasks complex tasks object recognition counting spatial relationships Visual Instruction Tuning VIT compositional complexity atomic capabilities complex capabilities benchmarks LLaVA-665k MMStar MM-Vet