The Same But Different: Structural Similarities and Differences in
  Multilingual Language Modeling
We employ new tools from mechanistic interpretability in order to ask whether
the internal structure of large language models (LLMs) shows correspondence to
the linguistic structures which underlie the languages on which they are
trained. In particular, we ask (1) when two languages employ the same
morphosyntactic processes, do LLMs handle them using shared internal circuitry?
and (2) when two languages require different morphosyntactic processes, do LLMs
handle them using different internal circuitry? Using English and Chinese
multilingual and monolingual models, we analyze the internal circuitry involved
in two tasks. We find evidence that models employ the same circuit to handle
the same syntactic process independently of the language in which it occurs,
and that this is the case even for monolingual models trained completely
independently. Moreover, we show that multilingual models employ
language-specific components (attention heads and feed-forward networks) when
needed to handle linguistic processes (e.g., morphological marking) that only
exist in some languages. Together, our results provide new insights into how
LLMs trade off between exploiting common structures and preserving linguistic
differences when tasked with modeling multiple languages simultaneously.
670e91997e9b2018da22717e
Ruochen Zhang
admin_assigned
2024-10-15T16:43:59.951Z
5fc54c33fa6eef7667a4d699
/avatars/24cce84812f724fdbcd950bbbae9b154.svg
Ruochen
user
ruochenz
670e91997e9b2018da22717f
Qinan Yu
admin_assigned
2024-10-15T16:44:06.598Z
66f577584b2e6a5c8e433942
/avatars/40ba38c3e8d4ca55fb7b15fc78aadccd.svg
Qinan Yu
user
qinanyu
670e91997e9b2018da227180
Matianyu Zang
admin_assigned
2024-10-15T16:44:18.957Z
66fa4f2a6e6db689a26f37d4
/avatars/327d7992cd69620cc173c4a9dff093f3.svg
Yuki Zang
user
matianyuzang
670e91997e9b2018da227181
Carsten Eickhoff
670e91997e9b2018da227182
Ellie Pavlick
The Same But Different: Structural Similarities and Differences in
  Multilingual Language Modeling
We employ new tools from mechanistic interpretability in order to ask whether
the internal structure of large language models (LLMs) shows correspondence to
the linguistic structures which underlie the languages on which they are
trained. In particular, we ask (1) when two languages employ the same
morphosyntactic processes, do LLMs handle them using shared internal circuitry?
and (2) when two languages require different morphosyntactic processes, do LLMs
handle them using different internal circuitry? Using English and Chinese
multilingual and monolingual models, we analyze the internal circuitry involved
in two tasks. We find evidence that models employ the same circuit to handle
the same syntactic process independently of the language in which it occurs,
and that this is the case even for monolingual models trained completely
independently. Moreover, we show that multilingual models employ
language-specific components (attention heads and feed-forward networks) when
needed to handle linguistic processes (e.g., morphological marking) that only
exist in some languages. Together, our results provide new insights into how
LLMs trade off between exploiting common structures and preserving linguistic
differences when tasked with modeling multiple languages simultaneously.
Large language models use shared internal structures for common syntactic processes across languages but incorporate language-specific components for unique linguistic features.
mechanistic interpretability large language models (LLMs) morphosyntactic processes internal circuitry attention heads feed-forward networks morphological marking