title: When Words Outperform Vision: VLMs Can Self-Improve Via Text-Only
    Training For Human-Centered Decision Making
summary: Embodied decision-making is fundamental for AI agents operating in real-world
  environments. While Visual Language Models (VLMs) have advanced this
  capability, they still struggle with complex decisions, particularly in
  human-centered situations that require deep reasoning about human needs and
  values. In this study, we systematically evaluate open-sourced VLMs on
  multimodal human-centered decision-making tasks. We find that LLMs receiving
  only textual descriptions unexpectedly outperform their VLM counterparts of
  similar scale that process actual images, suggesting that visual alignment may
  hinder VLM abilities. To address this challenge, we propose a novel text-only
  training approach with synthesized textual data. This method strengthens VLMs'
  language components and transfers the learned abilities to multimodal
  inference, eliminating the need for expensive image-text paired data.
  Furthermore, we show that VLMs can achieve substantial performance gains
  through self-improvement, using training data generated by their LLM
  counterparts rather than relying on larger teacher models like GPT-4. Our
  findings establish a more efficient and scalable approach to enhancing VLMs'
  human-centered decision-making capabilities, opening new avenues for optimizing
  VLMs through self-improvement mechanisms.
paper_authors._id: 67e35c3bf049c252c672b824
paper_authors.name: Zhe Hu
paper_authors._id: 67e35c3bf049c252c672b825
paper_authors.name: Jing Li
paper_authors._id: 67e35c3bf049c252c672b826
paper_authors.name: Yu Yin
paper_title: When Words Outperform Vision: VLMs Can Self-Improve Via Text-Only
    Training For Human-Centered Decision Making
paper_summary: Embodied decision-making is fundamental for AI agents operating in real-world
  environments. While Visual Language Models (VLMs) have advanced this
  capability, they still struggle with complex decisions, particularly in
  human-centered situations that require deep reasoning about human needs and
  values. In this study, we systematically evaluate open-sourced VLMs on
  multimodal human-centered decision-making tasks. We find that LLMs receiving
  only textual descriptions unexpectedly outperform their VLM counterparts of
  similar scale that process actual images, suggesting that visual alignment may
  hinder VLM abilities. To address this challenge, we propose a novel text-only
  training approach with synthesized textual data. This method strengthens VLMs'
  language components and transfers the learned abilities to multimodal
  inference, eliminating the need for expensive image-text paired data.
  Furthermore, we show that VLMs can achieve substantial performance gains
  through self-improvement, using training data generated by their LLM
  counterparts rather than relying on larger teacher models like GPT-4. Our
  findings establish a more efficient and scalable approach to enhancing VLMs'
  human-centered decision-making capabilities, opening new avenues for optimizing
  VLMs through self-improvement mechanisms.
paper_ai_summary: VLMs enhanced with synthesized textual data outperform traditional multimodal VLMs and can improve through self-generated training data, optimizing human-centered decision-making.
paper_ai_keywords: Visual Language Models (VLMs) Large Language Models (LLMs) multimodal human-centered decision-making tasks visual alignment text-only training synthesized textual data self-improvement training data generation