title: Towards Language Models That Can See: Computer Vision Through the LENS
    of Natural Language
summary: We propose LENS, a modular approach for tackling computer vision problems by
  leveraging the power of large language models (LLMs). Our system uses a
  language model to reason over outputs from a set of independent and highly
  descriptive vision modules that provide exhaustive information about an image.
  We evaluate the approach on pure computer vision settings such as zero- and
  few-shot object recognition, as well as on vision and language problems. LENS
  can be applied to any off-the-shelf LLM and we find that the LLMs with LENS
  perform highly competitively with much bigger and much more sophisticated
  systems, without any multimodal training whatsoever. We open-source our code at
  https://github.com/ContextualAI/lens and provide an interactive demo.
paper_authors._id: 649ce6f33e5ad160389de1a5
paper_authors.name: William Berrios
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-29T11:17:07.044Z
paper_authors._id: 62503f03815d0fd28f847c19
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1656275265222-62503f03815d0fd28f847c19.jpeg
paper_authors.fullname: William Berrios
paper_authors.type: user
paper_authors.user: will33am
paper_authors._id: 649ce6f33e5ad160389de1a6
paper_authors.name: Gautam Mittal
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-29T11:17:26.887Z
paper_authors._id: 6311b43205cc08a1408d5a78
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6311b43205cc08a1408d5a78/PMGdqvQ9H8LWnPc3L-KPp.jpeg
paper_authors.fullname: Gautam Mittal
paper_authors.type: user
paper_authors.user: gbm
paper_authors._id: 649ce6f33e5ad160389de1a7
paper_authors.name: Tristan Thrush
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-29T11:17:48.272Z
paper_authors._id: 61e9e3d4e2a95338e04c9f33
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61e9e3d4e2a95338e04c9f33/c3Pfr2LrD5Dbf0eFkScP6.png
paper_authors.fullname: Tristan Thrush
paper_authors.type: user
paper_authors.user: Tristan
paper_authors._id: 649ce6f33e5ad160389de1a8
paper_authors.name: Douwe Kiela
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-29T11:18:08.050Z
paper_authors._id: 61dc997715b47073db1620dc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1641847245435-61dc997715b47073db1620dc.jpeg
paper_authors.fullname: Douwe Kiela
paper_authors.type: user
paper_authors.user: douwekiela
paper_authors._id: 649ce6f33e5ad160389de1a9
paper_authors.name: Amanpreet Singh
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-19T09:36:07.606Z
paper_authors._id: 6230c6ecfd8b720a5648f6c4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1652253065548-6230c6ecfd8b720a5648f6c4.jpeg
paper_authors.fullname: Amanpreet Singh
paper_authors.type: user
paper_authors.user: aps
paper_title: Towards Language Models That Can See: Computer Vision Through the LENS
    of Natural Language
paper_summary: We propose LENS, a modular approach for tackling computer vision problems by
  leveraging the power of large language models (LLMs). Our system uses a
  language model to reason over outputs from a set of independent and highly
  descriptive vision modules that provide exhaustive information about an image.
  We evaluate the approach on pure computer vision settings such as zero- and
  few-shot object recognition, as well as on vision and language problems. LENS
  can be applied to any off-the-shelf LLM and we find that the LLMs with LENS
  perform highly competitively with much bigger and much more sophisticated
  systems, without any multimodal training whatsoever. We open-source our code at
  https://github.com/ContextualAI/lens and provide an interactive demo.
paper_ai_summary: LENS uses language models to reason over outputs from vision modules, achieving competitive performance in vision and vision-language tasks without multimodal training.
paper_ai_keywords: LENS large language models (LLMs) vision modules zero-shot object recognition few-shot object recognition multimodal training