Improving Visual Commonsense in Language Models via Multiple Image
  Generation
Commonsense reasoning is fundamentally based on multimodal knowledge.
However, existing large language models (LLMs) are primarily trained using
textual data only, limiting their ability to incorporate essential visual
information. In contrast, Visual Language Models, which excel at
visually-oriented tasks, often fail at non-visual tasks such as basic
commonsense reasoning. This divergence highlights a critical challenge - the
integration of robust visual understanding with foundational text-based
language reasoning. To this end, we introduce a method aimed at enhancing LLMs'
visual commonsense. Specifically, our method generates multiple images based on
the input text prompt and integrates these into the model's decision-making
process by mixing their prediction probabilities. To facilitate multimodal
grounded language modeling, we employ a late-fusion layer that combines the
projected visual features with the output of a pre-trained LLM conditioned on
text only. This late-fusion layer enables predictions based on comprehensive
image-text knowledge as well as text only when this is required. We evaluate
our approach using several visual commonsense reasoning tasks together with
traditional NLP tasks, including common sense reasoning and reading
comprehension. Our experimental results demonstrate significant superiority
over existing baselines. When applied to recent state-of-the-art LLMs (e.g.,
Llama3), we observe improvements not only in visual common sense but also in
traditional NLP benchmarks. Code and models are available under
https://github.com/guyyariv/vLMIG.
66756732c411cf3d84747370
Guy Yariv
claimed_verified
2024-06-21T11:47:41.536Z
646d239f4220471ca0c6471c
https://cdn-avatars.huggingface.co/v1/production/uploads/646d239f4220471ca0c6471c/sRwzko8XEUVCkeD7jXceH.jpeg
Guy Yariv
user
GuyYariv
66756732c411cf3d84747371
Idan Schwartz
66756732c411cf3d84747372
Yossi Adi
admin_assigned
2024-06-26T13:32:04.809Z
6481e135578646b5c2386728
https://cdn-avatars.huggingface.co/v1/production/uploads/6481e135578646b5c2386728/SPva4iNw0pORiCXD45cx9.jpeg
Yossi Adi
user
adiyoss
66756732c411cf3d84747373
Sagie Benaim
admin_assigned
2024-06-26T13:31:57.691Z
6345a9b9a8c2ff9f1377faab
/avatars/a5f2b999ef8b967b2af9f41afcd9d475.svg
Sagie Benaim
user
sagiebenaim
Improving Visual Commonsense in Language Models via Multiple Image
  Generation
Commonsense reasoning is fundamentally based on multimodal knowledge.
However, existing large language models (LLMs) are primarily trained using
textual data only, limiting their ability to incorporate essential visual
information. In contrast, Visual Language Models, which excel at
visually-oriented tasks, often fail at non-visual tasks such as basic
commonsense reasoning. This divergence highlights a critical challenge - the
integration of robust visual understanding with foundational text-based
language reasoning. To this end, we introduce a method aimed at enhancing LLMs'
visual commonsense. Specifically, our method generates multiple images based on
the input text prompt and integrates these into the model's decision-making
process by mixing their prediction probabilities. To facilitate multimodal
grounded language modeling, we employ a late-fusion layer that combines the
projected visual features with the output of a pre-trained LLM conditioned on
text only. This late-fusion layer enables predictions based on comprehensive
image-text knowledge as well as text only when this is required. We evaluate
our approach using several visual commonsense reasoning tasks together with
traditional NLP tasks, including common sense reasoning and reading
comprehension. Our experimental results demonstrate significant superiority
over existing baselines. When applied to recent state-of-the-art LLMs (e.g.,
Llama3), we observe improvements not only in visual common sense but also in
traditional NLP benchmarks. Code and models are available under
https://github.com/guyyariv/vLMIG.
A method enhances large language models' visual commonsense by generating images from text prompts and integrating them via a late-fusion layer with text-only predictions.
Visual Language Models large language models visual commonsense text-based language reasoning multimodal grounded language modeling late-fusion layer image-text knowledge common sense reasoning reading comprehension