title: Revisiting LRP: Positional Attribution as the Missing Ingredient for
    Transformer Explainability
summary: The development of effective explainability tools for Transformers is a
  crucial pursuit in deep learning research. One of the most promising approaches
  in this domain is Layer-wise Relevance Propagation (LRP), which propagates
  relevance scores backward through the network to the input space by
  redistributing activation values based on predefined rules. However, existing
  LRP-based methods for Transformer explainability entirely overlook a critical
  component of the Transformer architecture: its positional encoding (PE),
  resulting in violation of the conservation property, and the loss of an
  important and unique type of relevance, which is also associated with
  structural and positional features. To address this limitation, we reformulate
  the input space for Transformer explainability as a set of position-token
  pairs. This allows us to propose specialized theoretically-grounded LRP rules
  designed to propagate attributions across various positional encoding methods,
  including Rotary, Learnable, and Absolute PE. Extensive experiments with both
  fine-tuned classifiers and zero-shot foundation models, such as LLaMA 3,
  demonstrate that our method significantly outperforms the state-of-the-art in
  both vision and NLP explainability tasks. Our code is publicly available.
paper_authors._id: 683fe27c4f32bd7bbca087fc
paper_authors.name: Yarden Bakish
paper_authors._id: 683fe27c4f32bd7bbca087fd
paper_authors.name: Itamar Zimerman
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-06-05T07:32:51.961Z
paper_authors._id: 65376feed325b3f02fb92c69
paper_authors.avatarUrl: /avatars/e952918cf434d5302e9b1a404eccaf0e.svg
paper_authors.fullname: Itamar Zimerman
paper_authors.type: user
paper_authors.user: ItamarZ
paper_authors._id: 683fe27c4f32bd7bbca087fe
paper_authors.name: Hila Chefer
paper_authors._id: 683fe27c4f32bd7bbca087ff
paper_authors.name: Lior Wolf
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-06-05T07:33:06.122Z
paper_authors._id: 62f9f170363251ee40a08046
paper_authors.avatarUrl: /avatars/67ad459db44b605be0c1f6bd33363d6e.svg
paper_authors.fullname: Lior Wolf
paper_authors.type: user
paper_authors.user: liorwolf
paper_title: Revisiting LRP: Positional Attribution as the Missing Ingredient for
    Transformer Explainability
paper_summary: The development of effective explainability tools for Transformers is a
  crucial pursuit in deep learning research. One of the most promising approaches
  in this domain is Layer-wise Relevance Propagation (LRP), which propagates
  relevance scores backward through the network to the input space by
  redistributing activation values based on predefined rules. However, existing
  LRP-based methods for Transformer explainability entirely overlook a critical
  component of the Transformer architecture: its positional encoding (PE),
  resulting in violation of the conservation property, and the loss of an
  important and unique type of relevance, which is also associated with
  structural and positional features. To address this limitation, we reformulate
  the input space for Transformer explainability as a set of position-token
  pairs. This allows us to propose specialized theoretically-grounded LRP rules
  designed to propagate attributions across various positional encoding methods,
  including Rotary, Learnable, and Absolute PE. Extensive experiments with both
  fine-tuned classifiers and zero-shot foundation models, such as LLaMA 3,
  demonstrate that our method significantly outperforms the state-of-the-art in
  both vision and NLP explainability tasks. Our code is publicly available.
paper_ai_summary: A specialized LRP method for Transformer explainability considers positional encoding, improving relevance propagation and outperforming existing methods.
paper_ai_keywords: Layer-wise Relevance Propagation (LRP) Transformers positional encoding (PE) Rotary Learnable Absolute PE vision NLP explainability tasks