🔍 Today's pick in Interpretability & Analysis of LMs: AtP*: An efficient and scalable method for localizing LLM behaviour to components by J. Kramár T. Lieberum R. Shah 
text
🔍 Today's pick in Interpretability & Analysis of LMs: AtP*: An efficient and scalable method for localizing LLM behaviour to components by J. Kramár T. Lieberum R. Shah 
@NeelNanda
mention
NeelNanda
new_line
new_line
The attribution patching method (AtP) can provide fast and effective approximations of activation patching, requiring only two forward passes and one backward pass to estimate the contribution of all network components for a given prompt pair.
text
The attribution patching method (AtP) can provide fast and effective approximations of activation patching, requiring only two forward passes and one backward pass to estimate the contribution of all network components for a given prompt pair.
new_line
new_line
While previous work highlighted the effectiveness of attribution patching, authors identify two settings leading to false negatives using AtP:
text
While previous work highlighted the effectiveness of attribution patching, authors identify two settings leading to false negatives using AtP:
new_line
new_line
- When estimating the contribution of pre-activation components, if clean and noise inputs don’t lie in the same activation region, the first-order gradient approximation provided by the gradient leads to large errors (Fig 3).
text
- When estimating the contribution of pre-activation components, if clean and noise inputs don’t lie in the same activation region, the first-order gradient approximation provided by the gradient leads to large errors (Fig 3).
new_line
- When the sum of direct and indirect effects is close to 0, even small approximation errors introduced by nonlinearities can greatly  affect the estimated contribution.
text
- When the sum of direct and indirect effects is close to 0, even small approximation errors introduced by nonlinearities can greatly  affect the estimated contribution.
new_line
new_line
Authors propose two changes to the AtP method to mitigate such issues:
text
Authors propose two changes to the AtP method to mitigate such issues:
new_line
new_line
- recomputing the attention softmax for the selected component, and then taking a linear approximation to the remaining part of the model (QK Fix)
text
- recomputing the attention softmax for the selected component, and then taking a linear approximation to the remaining part of the model (QK Fix)
new_line
- Iteratively zeroing gradients at layers contributing to the indirect effects causing cancellation (GradDrop)
text
- Iteratively zeroing gradients at layers contributing to the indirect effects causing cancellation (GradDrop)
new_line
new_line
AtP and AtP* are compared across several patching settings for Pythia models, finding them effective while much less computationally expensive than other approaches. A new methodology is also proposed to estimate the magnitude of AtP* false negatives given a set of samples and desired confidence levels.
text
AtP and AtP* are compared across several patching settings for Pythia models, finding them effective while much less computationally expensive than other approaches. A new methodology is also proposed to estimate the magnitude of AtP* false negatives given a set of samples and desired confidence levels.
new_line
new_line
📄 Paper: 
text
📄 Paper: 
AtP*: An efficient and scalable method for localizing LLM behaviour to
  components (2403.00745)
https://huggingface.co/papers/2403.00745
2403.00745
paper
resource
https://huggingface.co/papers/2403.00745
new_line
new_line
🔍 All daily picks: 
text
🔍 All daily picks: 
https://huggingface.co/collections/gsarti/daily-picks-in-interpretability-and-analysis-of-lms-65ae3339949c5675d25de2f9
https://huggingface.co/collections/gsarti/daily-picks-in-interpretability-and-analysis-of-lms-65ae3339949c5675d25de2f9
link
new_line
🔍 Today's pick in Interpretability & Analysis of LMs: AtP*: An efficient and scalable method for localizing LLM behaviour to components by J. Kramár T. Lieberum R. Shah @NeelNanda

The attribution patching method (AtP) can provide fast and effective approximations of activation patching, requiring only two forward passes and one backward pass to estimate the contribution of all network components for a given prompt pair.

While previous work highlighted the effectiveness of attribution patching, authors identify two settings leading to false negatives using AtP:

- When estimating the contribution of pre-activation components, if clean and noise inputs don’t lie in the same activation region, the first-order gradient approximation provided by the gradient leads to large errors (Fig 3).
- When the sum of direct and indirect effects is close to 0, even small approximation errors introduced by nonlinearities can greatly  affect the estimated contribution.

Authors propose two changes to the AtP method to mitigate such issues:

- recomputing the attention softmax for the selected component, and then taking a linear approximation to the remaining part of the model (QK Fix)
- Iteratively zeroing gradients at layers contributing to the indirect effects causing cancellation (GradDrop)

AtP and AtP* are compared across several patching settings for Pythia models, finding them effective while much less computationally expensive than other approaches. A new methodology is also proposed to estimate the magnitude of AtP* false negatives given a set of samples and desired confidence levels.

📄 Paper: https://huggingface.co/papers/2403.00745

🔍 All daily picks: https://huggingface.co/collections/gsarti/daily-picks-in-interpretability-and-analysis-of-lms-65ae3339949c5675d25de2f9

❤️
osanseviero NeelNanda clem Kukedlc julien-c nirmalendu01
🤯
Kukedlc sbrandeis
64c2247b8eaab3b6286c2acc
/avatars/d364246314535c44cacd87a8e773525a.svg
nirmalendu prakash
nirmalendu01
user
5e7749883d77a72421292d07
https://cdn-avatars.huggingface.co/v1/production/uploads/5e7749883d77a72421292d07/M4AmBReZk_otxCIG3o0bL.jpeg
Gabriele Sarti
gsarti
user