title: The Same But Different: Structural Similarities and Differences in
    Multilingual Language Modeling
summary: We employ new tools from mechanistic interpretability in order to ask whether
  the internal structure of large language models (LLMs) shows correspondence to
  the linguistic structures which underlie the languages on which they are
  trained. In particular, we ask (1) when two languages employ the same
  morphosyntactic processes, do LLMs handle them using shared internal circuitry?
  and (2) when two languages require different morphosyntactic processes, do LLMs
  handle them using different internal circuitry? Using English and Chinese
  multilingual and monolingual models, we analyze the internal circuitry involved
  in two tasks. We find evidence that models employ the same circuit to handle
  the same syntactic process independently of the language in which it occurs,
  and that this is the case even for monolingual models trained completely
  independently. Moreover, we show that multilingual models employ
  language-specific components (attention heads and feed-forward networks) when
  needed to handle linguistic processes (e.g., morphological marking) that only
  exist in some languages. Together, our results provide new insights into how
  LLMs trade off between exploiting common structures and preserving linguistic
  differences when tasked with modeling multiple languages simultaneously.
paper_authors._id: 670e91997e9b2018da22717e
paper_authors.name: Ruochen Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-15T16:43:59.951Z
paper_authors._id: 5fc54c33fa6eef7667a4d699
paper_authors.avatarUrl: /avatars/24cce84812f724fdbcd950bbbae9b154.svg
paper_authors.fullname: Ruochen
paper_authors.type: user
paper_authors.user: ruochenz
paper_authors._id: 670e91997e9b2018da22717f
paper_authors.name: Qinan Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-15T16:44:06.598Z
paper_authors._id: 66f577584b2e6a5c8e433942
paper_authors.avatarUrl: /avatars/40ba38c3e8d4ca55fb7b15fc78aadccd.svg
paper_authors.fullname: Qinan Yu
paper_authors.type: user
paper_authors.user: qinanyu
paper_authors._id: 670e91997e9b2018da227180
paper_authors.name: Matianyu Zang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-15T16:44:18.957Z
paper_authors._id: 66fa4f2a6e6db689a26f37d4
paper_authors.avatarUrl: /avatars/327d7992cd69620cc173c4a9dff093f3.svg
paper_authors.fullname: Yuki Zang
paper_authors.type: user
paper_authors.user: matianyuzang
paper_authors._id: 670e91997e9b2018da227181
paper_authors.name: Carsten Eickhoff
paper_authors._id: 670e91997e9b2018da227182
paper_authors.name: Ellie Pavlick
paper_title: The Same But Different: Structural Similarities and Differences in
    Multilingual Language Modeling
paper_summary: We employ new tools from mechanistic interpretability in order to ask whether
  the internal structure of large language models (LLMs) shows correspondence to
  the linguistic structures which underlie the languages on which they are
  trained. In particular, we ask (1) when two languages employ the same
  morphosyntactic processes, do LLMs handle them using shared internal circuitry?
  and (2) when two languages require different morphosyntactic processes, do LLMs
  handle them using different internal circuitry? Using English and Chinese
  multilingual and monolingual models, we analyze the internal circuitry involved
  in two tasks. We find evidence that models employ the same circuit to handle
  the same syntactic process independently of the language in which it occurs,
  and that this is the case even for monolingual models trained completely
  independently. Moreover, we show that multilingual models employ
  language-specific components (attention heads and feed-forward networks) when
  needed to handle linguistic processes (e.g., morphological marking) that only
  exist in some languages. Together, our results provide new insights into how
  LLMs trade off between exploiting common structures and preserving linguistic
  differences when tasked with modeling multiple languages simultaneously.
paper_ai_keywords: mechanistic interpretability large language models (LLMs) morphosyntactic processes internal circuitry multilingual models monolingual models syntactic process language-specific components disconnection/dis and feed-forward networks morphological marking linguistic structures computational trade-offs