A Vision Check-up for Language Models
What does learning to model relationships between strings teach large
language models (LLMs) about the visual world? We systematically evaluate LLMs'
abilities to generate and recognize an assortment of visual concepts of
increasing complexity and then demonstrate how a preliminary visual
representation learning system can be trained using models of text. As language
models lack the ability to consume or output visual information as pixels, we
use code to represent images in our study. Although LLM-generated images do not
look like natural images, results on image generation and the ability of models
to correct these generated images indicate that precise modeling of strings can
teach language models about numerous aspects of the visual world. Furthermore,
experiments on self-supervised visual representation learning, utilizing images
generated with text models, highlight the potential to train vision models
capable of making semantic assessments of natural images using just LLMs.
659615e21ef0c62d514c669a
Pratyusha Sharma
claimed_verified
2024-05-03T07:17:39.246Z
6633cba69a2cd7d3299a287d
/avatars/d0f7465b5dacaf272ec61e608e7edde2.svg
Pratyusha Sharma
user
pratyushasharma
659615e21ef0c62d514c669b
Tamar Rott Shaham
659615e21ef0c62d514c669c
Manel Baradad
claimed_verified
2024-01-05T11:31:35.430Z
6261a12c18176506fab39eba
/avatars/e2b33b1fe6f43919dcb4df3410deda71.svg
Manel Baradad
user
manebaradad
659615e21ef0c62d514c669d
Stephanie Fu
659615e21ef0c62d514c669e
Adrian Rodriguez-Munoz
659615e21ef0c62d514c669f
Shivam Duggal
659615e21ef0c62d514c66a0
Phillip Isola
659615e21ef0c62d514c66a1
Antonio Torralba
A Vision Check-up for Language Models
What does learning to model relationships between strings teach large
language models (LLMs) about the visual world? We systematically evaluate LLMs'
abilities to generate and recognize an assortment of visual concepts of
increasing complexity and then demonstrate how a preliminary visual
representation learning system can be trained using models of text. As language
models lack the ability to consume or output visual information as pixels, we
use code to represent images in our study. Although LLM-generated images do not
look like natural images, results on image generation and the ability of models
to correct these generated images indicate that precise modeling of strings can
teach language models about numerous aspects of the visual world. Furthermore,
experiments on self-supervised visual representation learning, utilizing images
generated with text models, highlight the potential to train vision models
capable of making semantic assessments of natural images using just LLMs.
LLMs can learn aspects of the visual world through the precise modeling of string representations of images, which can also aid in training vision models for semantic assessments of natural images.
language models LLMs visual concepts image generation visual representation learning text models semantic assessments