Attention Heads of Large Language Models: A Survey
Since the advent of ChatGPT, Large Language Models (LLMs) have excelled in
various tasks but remain largely as black-box systems. Consequently, their
development relies heavily on data-driven approaches, limiting performance
enhancement through changes in internal architecture and reasoning pathways. As
a result, many researchers have begun exploring the potential internal
mechanisms of LLMs, aiming to identify the essence of their reasoning
bottlenecks, with most studies focusing on attention heads. Our survey aims to
shed light on the internal reasoning processes of LLMs by concentrating on the
interpretability and underlying mechanisms of attention heads. We first distill
the human thought process into a four-stage framework: Knowledge Recalling,
In-Context Identification, Latent Reasoning, and Expression Preparation. Using
this framework, we systematically review existing research to identify and
categorize the functions of specific attention heads. Furthermore, we summarize
the experimental methodologies used to discover these special heads, dividing
them into two categories: Modeling-Free methods and Modeling-Required methods.
Also, we outline relevant evaluation methods and benchmarks. Finally, we
discuss the limitations of current research and propose several potential
future directions. Our reference list is open-sourced at
https://github.com/IAAR-Shanghai/Awesome-Attention-Heads.
66da5f19247166c36547fd38
Zifan Zheng
claimed_verified
2024-09-06T07:06:25.188Z
656f47ba2f058b368c0b1611
https://cdn-avatars.huggingface.co/v1/production/uploads/656f47ba2f058b368c0b1611/mrmcmA8bxaDNUhuJQQ7T1.png
Zifan Zheng
user
fan2goa1
66da5f19247166c36547fd39
Yezhaohui Wang
admin_assigned
2024-09-13T17:55:35.519Z
662dd19f9e6d371ab71b91ce
https://cdn-avatars.huggingface.co/v1/production/uploads/662dd19f9e6d371ab71b91ce/mZBPw_Zs8ZlEFGlbekAoH.jpeg
Yezhaohui Wang
user
HaruTeru
66da5f19247166c36547fd3a
Yuxin Huang
admin_assigned
2024-09-06T09:07:22.127Z
669e891c49f46ef87af2457a
/avatars/2532729bb1cc9d52452ddb7852b41d72.svg
Yuxin Huang
user
saythe17
66da5f19247166c36547fd3b
Shichao Song
claimed_verified
2024-09-06T07:06:23.258Z
656f339a5273668d5b946b33
https://cdn-avatars.huggingface.co/v1/production/uploads/656f339a5273668d5b946b33/o2nBvQiOKKP5IfDmnpHP2.jpeg
Shichao Song
user
Ki-Seki
66da5f19247166c36547fd3c
Bo Tang
66da5f19247166c36547fd3d
Feiyu Xiong
66da5f19247166c36547fd3e
Zhiyu Li
admin_assigned
2024-09-06T09:08:20.895Z
62a155e615eeab266b2f2243
/avatars/e89ef156e73af028e3ce3664e6cb4e62.svg
Zhiyu Li
user
jimi888
Attention Heads of Large Language Models: A Survey
Since the advent of ChatGPT, Large Language Models (LLMs) have excelled in
various tasks but remain largely as black-box systems. Consequently, their
development relies heavily on data-driven approaches, limiting performance
enhancement through changes in internal architecture and reasoning pathways. As
a result, many researchers have begun exploring the potential internal
mechanisms of LLMs, aiming to identify the essence of their reasoning
bottlenecks, with most studies focusing on attention heads. Our survey aims to
shed light on the internal reasoning processes of LLMs by concentrating on the
interpretability and underlying mechanisms of attention heads. We first distill
the human thought process into a four-stage framework: Knowledge Recalling,
In-Context Identification, Latent Reasoning, and Expression Preparation. Using
this framework, we systematically review existing research to identify and
categorize the functions of specific attention heads. Furthermore, we summarize
the experimental methodologies used to discover these special heads, dividing
them into two categories: Modeling-Free methods and Modeling-Required methods.
Also, we outline relevant evaluation methods and benchmarks. Finally, we
discuss the limitations of current research and propose several potential
future directions. Our reference list is open-sourced at
https://github.com/IAAR-Shanghai/Awesome-Attention-Heads.
This survey explores the interpretability and underlying mechanisms of attention heads within Large Language Models to understand their reasoning processes.
Large Language Models LLMs attention heads Knowledge Recalling In-Context Identification Latent Reasoning Expression Preparation Modeling-Free methods Modeling-Required methods