Answers.Body: <p>This is an active research area, and we can be confident that we don't know the best techniques yet. There are overviews of the field <a href="https://en.wikipedia.org/wiki/Explainable_artificial_intelligence" rel="nofollow noreferrer">on Wikipedia</a>, <a href="https://ercim-news.ercim.eu/images/stories/EN134/EN134-web.pdf" rel="nofollow noreferrer">in this July 2023 special issue</a>, and <a href="https://www.alignmentforum.org/tag/interpretability-ml-and-ai" rel="nofollow noreferrer">in this tag on the AI alignment forum</a>. <a href="https://pauldeepakraj-r.medium.com/demystifying-the-black-box-a-deep-dive-into-llm-interpretability-971524966fdf" rel="nofollow noreferrer">This blog post</a> talks specifically about LLM interpretability.</p>
  <p>The most promising technique I know of for LLMs comes from Anthropic in their <a href="https://transformer-circuits.pub/2023/monosemantic-features/index.html" rel="nofollow noreferrer">Towards Monosemanticity paper</a>. Scott Alexander wrote <a href="https://www.astralcodexten.com/p/god-help-us-lets-try-to-understand" rel="nofollow noreferrer">an explainer with pointers to important background papers</a>. Unfortunately, it doesn't scale to Llama/Mistral levels yet.</p>
  <p>For more near-term applications, attention heatmaps should be scalable with some work. Here is a <a href="https://towardsdatascience.com/deconstructing-bert-part-2-visualizing-the-inner-workings-of-attention-60a16d86b5c1" rel="nofollow noreferrer">2019 blog post constructing one for BERT</a> and <a href="https://github.com/hnmr293/llama-viz" rel="nofollow noreferrer">a GitHub repo applying it to Llama</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-07T15:46:07.363Z
Answers.LastActivityDate: 2024-01-07T15:57:19.857Z
Answers.LastEditDate: 2024-01-07T15:57:19.857Z
Answers.UserDisplayName: Eponymous
Answers.Body: <p><a href="https://arxiv.org/abs/2311.17030" rel="nofollow noreferrer">Simple explanations of decision-making in LLMs are usually false.</a>
  Getting a correct explanation of how e.g. LLaMA-2-7b solves some task can be difficult. Here's my mental model of how tough it is in early 2024:</p>
  <ul>
  <li>could likely be done by reusing existing code over a few weeks of work: <a href="https://arxiv.org/abs/2211.00593" rel="nofollow noreferrer">indirect object identification</a>, <a href="https://arxiv.org/abs/2301.05217" rel="nofollow noreferrer">modular addition</a></li>
  <li>research project: factual Q&amp;A on a given topic, e.g. capitals of states, or birthdates of famous figures</li>
  <li>tough research project: anything involving composition of multiple facts stored in model's weights</li>
  <li>impossible with today's technology: e.g. solving coding problems, refusing to answer harmful requests</li>
  </ul>
  <p>My understanding is that, in early 2024, there does not exist a task where looking up how a 7B LLM does it is trivial, in the sense that you could get a confidently correct explanation in a few hours, using openly available tools.</p>
  <p>I recommend taking a look at <a href="https://neelnanda-io.github.io/TransformerLens/" rel="nofollow noreferrer">TransformerLens</a>; this <a href="https://github.com/neelnanda-io/TransformerLens" rel="nofollow noreferrer">repo</a> is used by many recent mechanistic interpretability. I guess (but have no inside info) that Google DeepMind and <a href="https://transformer-circuits.pub/2021/garcon/index.html" rel="nofollow noreferrer">Anthropic</a> have better tools internally.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-08T19:37:16.227Z
Answers.LastActivityDate: 2024-01-08T19:37:16.227Z
Answers.UserDisplayName: Daniel Paleka
Body: <p>I am currently working with large language models like llama and mistral, interested in techniques for making these models more explainable. I am looking for some tools or techniques which can help me with insights of how their models comes up with the predictions.</p>
  <p>Which are the tools and techniques for interpreting the decision-making process?</p>
  
Tags: <reference-request><large-language-models><explainable-ai>
Title: Which techniques are best suitable for explainable AI for LLM models
UserDisplayName: Hiren Namera
ThreadText: # Which techniques are best suitable for explainable AI for LLM models
  
  (Asked by: username_0 on 2024-01-07)
  
  I am currently working with large language models like llama and mistral, interested in techniques for making these models more explainable. I am looking for some tools or techniques which can help me with insights of how their models comes up with the predictions.
  
  Which are the tools and techniques for interpreting the decision-making process?
  
  **Tags:** reference-request, large-language-models, explainable-ai
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2024-01-07. Score: 2
  
  This is an active research area, and we can be confident that we don't know the best techniques yet. There are overviews of the field [on Wikipedia](https://en.wikipedia.org/wiki/Explainable_artificial_intelligence), [in this July 2023 special issue](https://ercim-news.ercim.eu/images/stories/EN134/EN134-web.pdf), and [in this tag on the AI alignment forum](https://www.alignmentforum.org/tag/interpretability-ml-and-ai). [This blog post](https://pauldeepakraj-r.medium.com/demystifying-the-black-box-a-deep-dive-into-llm-interpretability-971524966fdf) talks specifically about LLM interpretability.
  
  The most promising technique I know of for LLMs comes from Anthropic in their [Towards Monosemanticity paper](https://transformer-circuits.pub/2023/monosemantic-features/index.html). Scott Alexander wrote [an explainer with pointers to important background papers](https://www.astralcodexten.com/p/god-help-us-lets-try-to-understand). Unfortunately, it doesn't scale to Llama/Mistral levels yet.
  
  For more near-term applications, attention heatmaps should be scalable with some work. Here is a [2019 blog post constructing one for BERT](https://towardsdatascience.com/deconstructing-bert-part-2-visualizing-the-inner-workings-of-attention-60a16d86b5c1) and [a GitHub repo applying it to Llama](https://github.com/hnmr293/llama-viz).
  
  
  ---
  
  ### Answer by username_2 on 2024-01-08. Score: 2
  
  [Simple explanations of decision-making in LLMs are usually false.](https://arxiv.org/abs/2311.17030)
  Getting a correct explanation of how e.g. LLaMA-2-7b solves some task can be difficult. Here's my mental model of how tough it is in early 2024:
  
  * could likely be done by reusing existing code over a few weeks of work: [indirect object identification](https://arxiv.org/abs/2211.00593), [modular addition](https://arxiv.org/abs/2301.05217)
  * research project: factual Q&A on a given topic, e.g. capitals of states, or birthdates of famous figures
  * tough research project: anything involving composition of multiple facts stored in model's weights
  * impossible with today's technology: e.g. solving coding problems, refusing to answer harmful requests
  
  My understanding is that, in early 2024, there does not exist a task where looking up how a 7B LLM does it is trivial, in the sense that you could get a confidently correct explanation in a few hours, using openly available tools.
  
  I recommend taking a look at [TransformerLens](https://neelnanda-io.github.io/TransformerLens/); this [repo](https://github.com/neelnanda-io/TransformerLens) is used by many recent mechanistic interpretability. I guess (but have no inside info) that Google DeepMind and [Anthropic](https://transformer-circuits.pub/2021/garcon/index.html) have better tools internally.
  
  
  ---
  
  