Does Time Have Its Place? Temporal Heads: Where Language Models Recall
  Time-specific Information
While the ability of language models to elicit facts has been widely
investigated, how they handle temporally changing facts remains underexplored.
We discover Temporal Heads, specific attention heads primarily responsible for
processing temporal knowledge through circuit analysis. We confirm that these
heads are present across multiple models, though their specific locations may
vary, and their responses differ depending on the type of knowledge and its
corresponding years. Disabling these heads degrades the model's ability to
recall time-specific knowledge while maintaining its general capabilities
without compromising time-invariant and question-answering performances.
Moreover, the heads are activated not only numeric conditions ("In 2004") but
also textual aliases ("In the year ..."), indicating that they encode a
temporal dimension beyond simple numerical representation. Furthermore, we
expand the potential of our findings by demonstrating how temporal knowledge
can be edited by adjusting the values of these heads.
67b7fa96c3f48f8b3fc632fe
Yein Park
admin_assigned
2025-02-21T14:46:28.888Z
64e5c8e594aa0690321f6b29
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/IW5LRzmPcAM-dri8taMN7.png
Yein Park
user
P-YI
67b7fa96c3f48f8b3fc632ff
Chanwoong Yoon
admin_assigned
2025-02-21T14:46:35.045Z
66569dbaed45f790fbbebb83
/avatars/a4915e88d2bdff48cb30dd9972640d1e.svg
Chanwoong Yoon
user
cwyoon99
67b7fa96c3f48f8b3fc63300
Jungwoo Park
admin_assigned
2025-02-21T14:46:49.118Z
60f8435644e75317cc02ed51
/avatars/68b7fc077fe2bda6607b1c470add8140.svg
Jungwoo Park
user
affjljoo3581
67b7fa96c3f48f8b3fc63301
Minbyul Jeong
admin_assigned
2025-02-21T14:46:55.040Z
64587be872b60ae7a3817858
https://cdn-avatars.huggingface.co/v1/production/uploads/64587be872b60ae7a3817858/BbdOOxOCEzWTvEpkWp8MM.png
Minbyul Jeong
user
Minbyul
67b7fa96c3f48f8b3fc63302
Jaewoo Kang
Does Time Have Its Place? Temporal Heads: Where Language Models Recall
  Time-specific Information
While the ability of language models to elicit facts has been widely
investigated, how they handle temporally changing facts remains underexplored.
We discover Temporal Heads, specific attention heads primarily responsible for
processing temporal knowledge through circuit analysis. We confirm that these
heads are present across multiple models, though their specific locations may
vary, and their responses differ depending on the type of knowledge and its
corresponding years. Disabling these heads degrades the model's ability to
recall time-specific knowledge while maintaining its general capabilities
without compromising time-invariant and question-answering performances.
Moreover, the heads are activated not only numeric conditions ("In 2004") but
also textual aliases ("In the year ..."), indicating that they encode a
temporal dimension beyond simple numerical representation. Furthermore, we
expand the potential of our findings by demonstrating how temporal knowledge
can be edited by adjusting the values of these heads.
Temporal Heads in language models process and edit temporal knowledge through specific attention mechanisms, affecting recall without impacting general performance.
Temporal Heads attention heads temporal knowledge circuit analysis time-specific knowledge general capabilities time-invariant question-answering performances temporal dimension textual aliases numeric conditions