title: Do Vision and Language Models Share Concepts? A Vector Space Alignment
    Study
summary: Large-scale pretrained language models (LMs) are said to ``lack the ability
  to connect utterances to the world'' (Bender and Koller, 2020), because they do
  not have ``mental models of the world' '(Mitchell and Krakauer, 2023). If so,
  one would expect LM representations to be unrelated to representations induced
  by vision models. We present an empirical evaluation across four families of
  LMs (BERT, GPT-2, OPT and LLaMA-2) and three vision model architectures
  (ResNet, SegFormer, and MAE). Our experiments show that LMs partially converge
  towards representations isomorphic to those of vision models, subject to
  dispersion, polysemy and frequency. This has important implications for both
  multi-modal processing and the LM understanding debate (Mitchell and Krakauer,
  2023).
paper_authors._id: 668eef0c0b9a12807b071173
paper_authors.name: Jiaang Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-11T08:27:02.582Z
paper_authors._id: 6686658221f096c7fb4e0b24
paper_authors.avatarUrl: /avatars/581f5d56b393b95c74f38a06f6bb4f69.svg
paper_authors.fullname: Jiaang Li
paper_authors.type: user
paper_authors.user: jaagli
paper_authors._id: 668eef0c0b9a12807b071174
paper_authors.name: Yova Kementchedjhieva
paper_authors._id: 668eef0c0b9a12807b071175
paper_authors.name: Constanza Fierro
paper_authors._id: 668eef0c0b9a12807b071176
paper_authors.name: Anders Søgaard
paper_title: Do Vision and Language Models Share Concepts? A Vector Space Alignment
    Study
paper_summary: Large-scale pretrained language models (LMs) are said to ``lack the ability
  to connect utterances to the world'' (Bender and Koller, 2020), because they do
  not have ``mental models of the world' '(Mitchell and Krakauer, 2023). If so,
  one would expect LM representations to be unrelated to representations induced
  by vision models. We present an empirical evaluation across four families of
  LMs (BERT, GPT-2, OPT and LLaMA-2) and three vision model architectures
  (ResNet, SegFormer, and MAE). Our experiments show that LMs partially converge
  towards representations isomorphic to those of vision models, subject to
  dispersion, polysemy and frequency. This has important implications for both
  multi-modal processing and the LM understanding debate (Mitchell and Krakauer,
  2023).
paper_ai_summary: Large-scale pretrained language models partially converge to representations similar to those of vision models, impacting multi-modal processing and the debate on language model understanding.
paper_ai_keywords: pretrained language models mental models representations vision models BERT GPT-2 OPT LLaMA-2 ResNet SegFormer MAE isomorphic representations dispersion polysemy frequency