title: Does Time Have Its Place? Temporal Heads: Where Language Models Recall
    Time-specific Information
summary: While the ability of language models to elicit facts has been widely
  investigated, how they handle temporally changing facts remains underexplored.
  We discover Temporal Heads, specific attention heads primarily responsible for
  processing temporal knowledge through circuit analysis. We confirm that these
  heads are present across multiple models, though their specific locations may
  vary, and their responses differ depending on the type of knowledge and its
  corresponding years. Disabling these heads degrades the model's ability to
  recall time-specific knowledge while maintaining its general capabilities
  without compromising time-invariant and question-answering performances.
  Moreover, the heads are activated not only numeric conditions ("In 2004") but
  also textual aliases ("In the year ..."), indicating that they encode a
  temporal dimension beyond simple numerical representation. Furthermore, we
  expand the potential of our findings by demonstrating how temporal knowledge
  can be edited by adjusting the values of these heads.
paper_authors._id: 67b7fa96c3f48f8b3fc632fe
paper_authors.name: Yein Park
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-21T14:46:28.888Z
paper_authors._id: 64e5c8e594aa0690321f6b29
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/IW5LRzmPcAM-dri8taMN7.png
paper_authors.fullname: Yein Park
paper_authors.type: user
paper_authors.user: P-YI
paper_authors._id: 67b7fa96c3f48f8b3fc632ff
paper_authors.name: Chanwoong Yoon
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-21T14:46:35.045Z
paper_authors._id: 66569dbaed45f790fbbebb83
paper_authors.avatarUrl: /avatars/a4915e88d2bdff48cb30dd9972640d1e.svg
paper_authors.fullname: Chanwoong Yoon
paper_authors.type: user
paper_authors.user: cwyoon99
paper_authors._id: 67b7fa96c3f48f8b3fc63300
paper_authors.name: Jungwoo Park
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-21T14:46:49.118Z
paper_authors._id: 60f8435644e75317cc02ed51
paper_authors.avatarUrl: /avatars/68b7fc077fe2bda6607b1c470add8140.svg
paper_authors.fullname: Jungwoo Park
paper_authors.type: user
paper_authors.user: affjljoo3581
paper_authors._id: 67b7fa96c3f48f8b3fc63301
paper_authors.name: Minbyul Jeong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-21T14:46:55.040Z
paper_authors._id: 64587be872b60ae7a3817858
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64587be872b60ae7a3817858/BbdOOxOCEzWTvEpkWp8MM.png
paper_authors.fullname: Minbyul Jeong
paper_authors.type: user
paper_authors.user: Minbyul
paper_authors._id: 67b7fa96c3f48f8b3fc63302
paper_authors.name: Jaewoo Kang
paper_title: Does Time Have Its Place? Temporal Heads: Where Language Models Recall
    Time-specific Information
paper_summary: While the ability of language models to elicit facts has been widely
  investigated, how they handle temporally changing facts remains underexplored.
  We discover Temporal Heads, specific attention heads primarily responsible for
  processing temporal knowledge through circuit analysis. We confirm that these
  heads are present across multiple models, though their specific locations may
  vary, and their responses differ depending on the type of knowledge and its
  corresponding years. Disabling these heads degrades the model's ability to
  recall time-specific knowledge while maintaining its general capabilities
  without compromising time-invariant and question-answering performances.
  Moreover, the heads are activated not only numeric conditions ("In 2004") but
  also textual aliases ("In the year ..."), indicating that they encode a
  temporal dimension beyond simple numerical representation. Furthermore, we
  expand the potential of our findings by demonstrating how temporal knowledge
  can be edited by adjusting the values of these heads.
paper_ai_keywords: Temporal Heads attention heads circuit analysis temporal knowledge time-specific knowledge time-invariant