title: The Semantic Hub Hypothesis: Language Models Share Semantic
    Representations Across Languages and Modalities
summary: Modern language models can process inputs across diverse languages and
  modalities. We hypothesize that models acquire this capability through learning
  a shared representation space across heterogeneous data types (e.g., different
  languages and modalities), which places semantically similar inputs near one
  another, even if they are from different modalities/languages. We term this the
  semantic hub hypothesis, following the hub-and-spoke model from neuroscience
  (Patterson et al., 2007) which posits that semantic knowledge in the human
  brain is organized through a transmodal semantic "hub" which integrates
  information from various modality-specific "spokes" regions. We first show that
  model representations for semantically equivalent inputs in different languages
  are similar in the intermediate layers, and that this space can be interpreted
  using the model's dominant pretraining language via the logit lens. This
  tendency extends to other data types, including arithmetic expressions, code,
  and visual/audio inputs. Interventions in the shared representation space in
  one data type also predictably affect model outputs in other data types,
  suggesting that this shared representations space is not simply a vestigial
  byproduct of large-scale training on broad data, but something that is actively
  utilized by the model during input processing.
paper_authors._id: 67321c68254c0b2144d7a068
paper_authors.name: Zhaofeng Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-14T08:01:54.054Z
paper_authors._id: 6351712b40dffad651f128c7
paper_authors.avatarUrl: /avatars/87708c86c1baef548ef556f5d32dca71.svg
paper_authors.fullname: Zhaofeng Wu
paper_authors.type: user
paper_authors.user: ZhaofengWu
paper_authors._id: 67321c68254c0b2144d7a069
paper_authors.name: Xinyan Velocity Yu
paper_authors._id: 67321c68254c0b2144d7a06a
paper_authors.name: Dani Yogatama
paper_authors._id: 67321c68254c0b2144d7a06b
paper_authors.name: Jiasen Lu
paper_authors._id: 67321c68254c0b2144d7a06c
paper_authors.name: Yoon Kim
paper_title: The Semantic Hub Hypothesis: Language Models Share Semantic
    Representations Across Languages and Modalities
paper_summary: Modern language models can process inputs across diverse languages and
  modalities. We hypothesize that models acquire this capability through learning
  a shared representation space across heterogeneous data types (e.g., different
  languages and modalities), which places semantically similar inputs near one
  another, even if they are from different modalities/languages. We term this the
  semantic hub hypothesis, following the hub-and-spoke model from neuroscience
  (Patterson et al., 2007) which posits that semantic knowledge in the human
  brain is organized through a transmodal semantic "hub" which integrates
  information from various modality-specific "spokes" regions. We first show that
  model representations for semantically equivalent inputs in different languages
  are similar in the intermediate layers, and that this space can be interpreted
  using the model's dominant pretraining language via the logit lens. This
  tendency extends to other data types, including arithmetic expressions, code,
  and visual/audio inputs. Interventions in the shared representation space in
  one data type also predictably affect model outputs in other data types,
  suggesting that this shared representations space is not simply a vestigial
  byproduct of large-scale training on broad data, but something that is actively
  utilized by the model during input processing.
paper_ai_summary: Language models develop a shared representation space that organizes semantically similar inputs from diverse languages and modalities, facilitating cross-modal and cross-lingual understanding and processing.
paper_ai_keywords: semantic hub hypothesis hub-and-spoke model semantic representation space pretraining language logit lens arithmetic expressions code visual/audio inputs