title: Enhancing Cognition and Explainability of Multimodal Foundation Models
    with Self-Synthesized Data
summary: Large multimodal models (LMMs) have shown impressive capabilities in a wide
  range of visual tasks. However, they often struggle with fine-grained visual
  reasoning, failing to identify domain-specific objectives and provide
  justifiable explanations for their predictions. To address this, we propose a
  novel visual rejection sampling framework to improve the cognition and
  explainability of LMMs using self-synthesized data. Specifically, visual
  fine-tuning requires images, queries, and target answers. Our approach begins
  by synthesizing interpretable answers that include human-verifiable visual
  features. These features are based on expert-defined concepts, carefully
  selected based on their alignment with the image content. After each round of
  fine-tuning, we apply a reward model-free filtering mechanism to select the
  highest-quality interpretable answers for the next round of tuning. This
  iterative process of data synthesis and fine-tuning progressively improves the
  model's ability to generate accurate and reasonable explanations. Experimental
  results demonstrate the effectiveness of our method in improving both the
  accuracy and explainability of specialized visual classification tasks.
paper_authors._id: 67b87e3a346553e4006bf37c
paper_authors.name: Yucheng Shi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-21T15:14:19.237Z
paper_authors._id: 64beb6b6140491ca9f803ebf
paper_authors.avatarUrl: /avatars/0daa2e813a13668b8b708cd8c12763d9.svg
paper_authors.fullname: Yucheng SHi
paper_authors.type: user
paper_authors.user: YuchengShi
paper_authors._id: 67b87e3a346553e4006bf37d
paper_authors.name: Quanzheng Li
paper_authors._id: 67b87e3a346553e4006bf37e
paper_authors.name: Jin Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-21T15:15:13.205Z
paper_authors._id: 6409cde91ee054d66a66c817
paper_authors.avatarUrl: /avatars/ed18aa9902760a7ad6e9c5789b26dbe3.svg
paper_authors.fullname: jin sun
paper_authors.type: user
paper_authors.user: jinsun
paper_authors._id: 67b87e3a346553e4006bf37f
paper_authors.name: Xiang Li
paper_authors._id: 67b87e3a346553e4006bf380
paper_authors.name: Ninghao Liu
paper_title: Enhancing Cognition and Explainability of Multimodal Foundation Models
    with Self-Synthesized Data
paper_summary: Large multimodal models (LMMs) have shown impressive capabilities in a wide
  range of visual tasks. However, they often struggle with fine-grained visual
  reasoning, failing to identify domain-specific objectives and provide
  justifiable explanations for their predictions. To address this, we propose a
  novel visual rejection sampling framework to improve the cognition and
  explainability of LMMs using self-synthesized data. Specifically, visual
  fine-tuning requires images, queries, and target answers. Our approach begins
  by synthesizing interpretable answers that include human-verifiable visual
  features. These features are based on expert-defined concepts, carefully
  selected based on their alignment with the image content. After each round of
  fine-tuning, we apply a reward model-free filtering mechanism to select the
  highest-quality interpretable answers for the next round of tuning. This
  iterative process of data synthesis and fine-tuning progressively improves the
  model's ability to generate accurate and reasonable explanations. Experimental
  results demonstrate the effectiveness of our method in improving both the
  accuracy and explainability of specialized visual classification tasks.
paper_ai_keywords: Large multimodal models (LMMs) visual rejection sampling fine-grained visual reasoning self-synthesized data visual fine-tuning interpretable answers human-verifiable visual features expert-defined concepts reward model-free filtering mechanism specialized visual classification tasks