On Relation-Specific Neurons in Large Language Models
In large language models (LLMs), certain neurons can store distinct pieces of
knowledge learned during pretraining. While knowledge typically appears as a
combination of relations and entities, it remains unclear whether some neurons
focus on a relation itself -- independent of any entity. We hypothesize such
neurons detect a relation in the input text and guide generation involving such
a relation. To investigate this, we study the Llama-2 family on a chosen set of
relations with a statistics-based method. Our experiments demonstrate the
existence of relation-specific neurons. We measure the effect of selectively
deactivating candidate neurons specific to relation r on the LLM's ability to
handle (1) facts whose relation is r and (2) facts whose relation is a
different relation r' neq r. With respect to their capacity for encoding
relation information, we give evidence for the following three properties of
relation-specific neurons. (i) Neuron cumulativity. The neurons for
r present a cumulative effect so that deactivating a larger portion of them
results in the degradation of more facts in r. (ii) Neuron
versatility. Neurons can be shared across multiple closely related as well as
less related relations. Some relation neurons transfer across languages.
(iii) Neuron interference. Deactivating neurons specific to one
relation can improve LLM generation performance for facts of other relations.
We will make our code publicly available at
https://github.com/cisnlp/relation-specific-neurons.
67bf1808b91e7e6477d92c1e
Yihong Liu
claimed_verified
2025-02-28T15:14:48.351Z
653f7e569e84d1e8b6a66e70
/avatars/24eaa6434508a162c349aebfc51990ff.svg
Yihong Liu
user
yihongLiu
67bf1808b91e7e6477d92c1f
Runsheng Chen
admin_assigned
2025-02-28T15:16:28.041Z
63629b9f2a84d82a8c8feb32
/avatars/8484b5bf8311b28249757729b1ce80f8.svg
Chen
user
Runsheng
67bf1808b91e7e6477d92c20
Lea Hirlimann
admin_assigned
2025-02-28T15:16:18.398Z
658559148615630cb3ec5b6b
/avatars/dd804ca277e6b19903bb550cc167ba4a.svg
Lea Hirlimann
user
hirlimann
67bf1808b91e7e6477d92c21
Ahmad Dawar Hakimi
admin_assigned
2025-02-28T15:16:11.693Z
62502669d2d191ac43320ade
/avatars/7997e9b2012059edb22b745c3b737481.svg
Ahmad Dawar Hakimi
user
adhakimi
67bf1808b91e7e6477d92c22
Mingyang Wang
67bf1808b91e7e6477d92c23
Amir Hossein Kargaran
claimed_verified
2025-02-26T15:37:07.932Z
61bf84c8ca59d6d196a1b4e8
https://cdn-avatars.huggingface.co/v1/production/uploads/61bf84c8ca59d6d196a1b4e8/L_NvUwlMYcye9X35z6f7e.jpeg
Amir Hossein Kargaran
user
kargaranamir
67bf1808b91e7e6477d92c24
Sascha Rothe
67bf1808b91e7e6477d92c25
François Yvon
admin_assigned
2025-02-28T15:16:57.343Z
62ab10f04bd2ebf5dbad205c
/avatars/65356b3b057159cc67a86efb26b53486.svg
François Yvon
user
fyvo
67bf1808b91e7e6477d92c26
Hinrich Schütze
On Relation-Specific Neurons in Large Language Models
In large language models (LLMs), certain neurons can store distinct pieces of
knowledge learned during pretraining. While knowledge typically appears as a
combination of relations and entities, it remains unclear whether some neurons
focus on a relation itself -- independent of any entity. We hypothesize such
neurons detect a relation in the input text and guide generation involving such
a relation. To investigate this, we study the Llama-2 family on a chosen set of
relations with a statistics-based method. Our experiments demonstrate the
existence of relation-specific neurons. We measure the effect of selectively
deactivating candidate neurons specific to relation r on the LLM's ability to
handle (1) facts whose relation is r and (2) facts whose relation is a
different relation r' neq r. With respect to their capacity for encoding
relation information, we give evidence for the following three properties of
relation-specific neurons. (i) Neuron cumulativity. The neurons for
r present a cumulative effect so that deactivating a larger portion of them
results in the degradation of more facts in r. (ii) Neuron
versatility. Neurons can be shared across multiple closely related as well as
less related relations. Some relation neurons transfer across languages.
(iii) Neuron interference. Deactivating neurons specific to one
relation can improve LLM generation performance for facts of other relations.
We will make our code publicly available at
https://github.com/cisnlp/relation-specific-neurons.
Studies on the Llama-2 family reveal relation-specific neurons that cumulatively encode relation information, are versatile across related relations, and can interfere with each other, impacting LLM performance.
large language models LLMs relation-specific neurons pretraining knowledge storage relation detection neuron cumulativity neuron versatility neuron interference