title: A Vision Check-up for Language Models
summary: What does learning to model relationships between strings teach large
  language models (LLMs) about the visual world? We systematically evaluate LLMs'
  abilities to generate and recognize an assortment of visual concepts of
  increasing complexity and then demonstrate how a preliminary visual
  representation learning system can be trained using models of text. As language
  models lack the ability to consume or output visual information as pixels, we
  use code to represent images in our study. Although LLM-generated images do not
  look like natural images, results on image generation and the ability of models
  to correct these generated images indicate that precise modeling of strings can
  teach language models about numerous aspects of the visual world. Furthermore,
  experiments on self-supervised visual representation learning, utilizing images
  generated with text models, highlight the potential to train vision models
  capable of making semantic assessments of natural images using just LLMs.
paper_authors._id: 659615e21ef0c62d514c669a
paper_authors.name: Pratyusha Sharma
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-03T07:17:39.246Z
paper_authors._id: 6633cba69a2cd7d3299a287d
paper_authors.avatarUrl: /avatars/d0f7465b5dacaf272ec61e608e7edde2.svg
paper_authors.fullname: Pratyusha Sharma
paper_authors.type: user
paper_authors.user: pratyushasharma
paper_authors._id: 659615e21ef0c62d514c669b
paper_authors.name: Tamar Rott Shaham
paper_authors._id: 659615e21ef0c62d514c669c
paper_authors.name: Manel Baradad
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-05T11:31:35.430Z
paper_authors._id: 6261a12c18176506fab39eba
paper_authors.avatarUrl: /avatars/e2b33b1fe6f43919dcb4df3410deda71.svg
paper_authors.fullname: Manel Baradad
paper_authors.type: user
paper_authors.user: manebaradad
paper_authors._id: 659615e21ef0c62d514c669d
paper_authors.name: Stephanie Fu
paper_authors._id: 659615e21ef0c62d514c669e
paper_authors.name: Adrian Rodriguez-Munoz
paper_authors._id: 659615e21ef0c62d514c669f
paper_authors.name: Shivam Duggal
paper_authors._id: 659615e21ef0c62d514c66a0
paper_authors.name: Phillip Isola
paper_authors._id: 659615e21ef0c62d514c66a1
paper_authors.name: Antonio Torralba
paper_title: A Vision Check-up for Language Models
paper_summary: What does learning to model relationships between strings teach large
  language models (LLMs) about the visual world? We systematically evaluate LLMs'
  abilities to generate and recognize an assortment of visual concepts of
  increasing complexity and then demonstrate how a preliminary visual
  representation learning system can be trained using models of text. As language
  models lack the ability to consume or output visual information as pixels, we
  use code to represent images in our study. Although LLM-generated images do not
  look like natural images, results on image generation and the ability of models
  to correct these generated images indicate that precise modeling of strings can
  teach language models about numerous aspects of the visual world. Furthermore,
  experiments on self-supervised visual representation learning, utilizing images
  generated with text models, highlight the potential to train vision models
  capable of making semantic assessments of natural images using just LLMs.
paper_ai_keywords: Tensor-Web Compiling Co-Design Visual representation learning Language Models (LLMs) Natural images Semantic assessments