title: Improving Visual Commonsense in Language Models via Multiple Image
    Generation
summary: Commonsense reasoning is fundamentally based on multimodal knowledge.
  However, existing large language models (LLMs) are primarily trained using
  textual data only, limiting their ability to incorporate essential visual
  information. In contrast, Visual Language Models, which excel at
  visually-oriented tasks, often fail at non-visual tasks such as basic
  commonsense reasoning. This divergence highlights a critical challenge - the
  integration of robust visual understanding with foundational text-based
  language reasoning. To this end, we introduce a method aimed at enhancing LLMs'
  visual commonsense. Specifically, our method generates multiple images based on
  the input text prompt and integrates these into the model's decision-making
  process by mixing their prediction probabilities. To facilitate multimodal
  grounded language modeling, we employ a late-fusion layer that combines the
  projected visual features with the output of a pre-trained LLM conditioned on
  text only. This late-fusion layer enables predictions based on comprehensive
  image-text knowledge as well as text only when this is required. We evaluate
  our approach using several visual commonsense reasoning tasks together with
  traditional NLP tasks, including common sense reasoning and reading
  comprehension. Our experimental results demonstrate significant superiority
  over existing baselines. When applied to recent state-of-the-art LLMs (e.g.,
  Llama3), we observe improvements not only in visual common sense but also in
  traditional NLP benchmarks. Code and models are available under
  https://github.com/guyyariv/vLMIG.
paper_authors._id: 66756732c411cf3d84747370
paper_authors.name: Guy Yariv
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-21T11:47:41.536Z
paper_authors._id: 646d239f4220471ca0c6471c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/646d239f4220471ca0c6471c/sRwzko8XEUVCkeD7jXceH.jpeg
paper_authors.fullname: Guy Yariv
paper_authors.type: user
paper_authors.user: GuyYariv
paper_authors._id: 66756732c411cf3d84747371
paper_authors.name: Idan Schwartz
paper_authors._id: 66756732c411cf3d84747372
paper_authors.name: Yossi Adi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-26T13:32:04.809Z
paper_authors._id: 6481e135578646b5c2386728
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6481e135578646b5c2386728/SPva4iNw0pORiCXD45cx9.jpeg
paper_authors.fullname: Yossi Adi
paper_authors.type: user
paper_authors.user: adiyoss
paper_authors._id: 66756732c411cf3d84747373
paper_authors.name: Sagie Benaim
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-26T13:31:57.691Z
paper_authors._id: 6345a9b9a8c2ff9f1377faab
paper_authors.avatarUrl: /avatars/a5f2b999ef8b967b2af9f41afcd9d475.svg
paper_authors.fullname: Sagie Benaim
paper_authors.type: user
paper_authors.user: sagiebenaim
paper_title: Improving Visual Commonsense in Language Models via Multiple Image
    Generation
paper_summary: Commonsense reasoning is fundamentally based on multimodal knowledge.
  However, existing large language models (LLMs) are primarily trained using
  textual data only, limiting their ability to incorporate essential visual
  information. In contrast, Visual Language Models, which excel at
  visually-oriented tasks, often fail at non-visual tasks such as basic
  commonsense reasoning. This divergence highlights a critical challenge - the
  integration of robust visual understanding with foundational text-based
  language reasoning. To this end, we introduce a method aimed at enhancing LLMs'
  visual commonsense. Specifically, our method generates multiple images based on
  the input text prompt and integrates these into the model's decision-making
  process by mixing their prediction probabilities. To facilitate multimodal
  grounded language modeling, we employ a late-fusion layer that combines the
  projected visual features with the output of a pre-trained LLM conditioned on
  text only. This late-fusion layer enables predictions based on comprehensive
  image-text knowledge as well as text only when this is required. We evaluate
  our approach using several visual commonsense reasoning tasks together with
  traditional NLP tasks, including common sense reasoning and reading
  comprehension. Our experimental results demonstrate significant superiority
  over existing baselines. When applied to recent state-of-the-art LLMs (e.g.,
  Llama3), we observe improvements not only in visual common sense but also in
  traditional NLP benchmarks. Code and models are available under
  https://github.com/guyyariv/vLMIG.
paper_ai_summary: A method enhances large language models' visual commonsense by generating images from text prompts and integrating them via a late-fusion layer with text-only predictions.
paper_ai_keywords: Visual Language Models large language models visual commonsense text-based language reasoning multimodal grounded language modeling late-fusion layer image-text knowledge common sense reasoning reading comprehension