How does Alignment Enhance LLMs' Multilingual Capabilities? A Language
  Neurons Perspective
Multilingual Alignment is an effective and representative paradigm to enhance
LLMs' multilingual capabilities, which transfers the capabilities from the
high-resource languages to the low-resource languages. Meanwhile, some
researches on language-specific neurons reveal that there are language-specific
neurons that are selectively activated in LLMs when processing different
languages. This provides a new perspective to analyze and understand LLMs'
mechanisms more specifically in multilingual scenarios. In this work, we
propose a new finer-grained neuron identification algorithm, which detects
language neurons~(including language-specific neurons and language-related
neurons) and language-agnostic neurons. Furthermore, based on the
distributional characteristics of different types of neurons, we divide the
LLMs' internal process for multilingual inference into four parts: (1)
multilingual understanding, (2) shared semantic space reasoning, (3)
multilingual output space transformation, and (4) vocabulary space outputting.
Additionally, we systematically analyze the models before and after alignment
with a focus on different types of neurons. We also analyze the phenomenon of
''Spontaneous Multilingual Alignment''. Overall, our work conducts a
comprehensive investigation based on different types of neurons, providing
empirical results and valuable insights for better understanding multilingual
alignment and multilingual capabilities of LLMs.
68369d9f8a36b9fa7f340c86
Shimao Zhang
claimed_verified
2025-05-28T08:56:40.548Z
65080dc63fc966d1bbba485d
/avatars/347890233f2316e7f7a04d652b2378bb.svg
Shimao Zhang
user
Shimao-Zhang
68369d9f8a36b9fa7f340c87
Zhejian Lai
claimed_verified
2025-05-28T08:56:38.385Z
643525ea0b30bd434ea15363
https://cdn-avatars.huggingface.co/v1/production/uploads/643525ea0b30bd434ea15363/7sAzllfWUPtt68NY1gDLj.png
Jackie Lai
user
DreamW1ngs
68369d9f8a36b9fa7f340c88
Xiang Liu
claimed_verified
2025-05-28T12:12:17.068Z
65cc78118ebd392213fc9e16
/avatars/859735d5abdb6f2c6f26bd8db034a1ce.svg
Xiang Liu
user
VincentLx
68369d9f8a36b9fa7f340c89
Shuaijie She
68369d9f8a36b9fa7f340c8a
Xiao Liu
claimed_verified
2025-05-29T20:18:53.657Z
63fb6e281b4b1bd4e7ffc5be
https://cdn-avatars.huggingface.co/v1/production/uploads/1677422062937-noauth.jpeg
Xiao Liu
user
lx865712528
68369d9f8a36b9fa7f340c8b
Yeyun Gong
68369d9f8a36b9fa7f340c8c
Shujian Huang
68369d9f8a36b9fa7f340c8d
Jiajun Chen
How does Alignment Enhance LLMs' Multilingual Capabilities? A Language
  Neurons Perspective
Multilingual Alignment is an effective and representative paradigm to enhance
LLMs' multilingual capabilities, which transfers the capabilities from the
high-resource languages to the low-resource languages. Meanwhile, some
researches on language-specific neurons reveal that there are language-specific
neurons that are selectively activated in LLMs when processing different
languages. This provides a new perspective to analyze and understand LLMs'
mechanisms more specifically in multilingual scenarios. In this work, we
propose a new finer-grained neuron identification algorithm, which detects
language neurons~(including language-specific neurons and language-related
neurons) and language-agnostic neurons. Furthermore, based on the
distributional characteristics of different types of neurons, we divide the
LLMs' internal process for multilingual inference into four parts: (1)
multilingual understanding, (2) shared semantic space reasoning, (3)
multilingual output space transformation, and (4) vocabulary space outputting.
Additionally, we systematically analyze the models before and after alignment
with a focus on different types of neurons. We also analyze the phenomenon of
''Spontaneous Multilingual Alignment''. Overall, our work conducts a
comprehensive investigation based on different types of neurons, providing
empirical results and valuable insights for better understanding multilingual
alignment and multilingual capabilities of LLMs.
The research proposes a finer-grained neuron identification algorithm for detecting language-specific and language-agnostic neurons in LLMs, and investigates the impact on multilingual alignment and capabilities through analysis of multilingual understanding, shared semantic reasoning, multilingual output transformation, and vocabulary space outputting.
multilingual alignment language-specific neurons language-agnostic neurons shared semantic space multilingual output space vocabulary space neuron identification algorithm spontaneous multilingual alignment