Body: <p>Although the multi-headed attention block of the transformer allows the model to be more expressive (and therefore perform better), it is remarkably more difficult to decompose and therefore to explain (in terms of model transparency).</p>
  <p>Would an attentive RNN be considered a better architecture w.r.t. model transparency and attention-based explainability?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-04-17T23:14:33.437Z
Comments.Text: Interesting question, but I think it's not easy to answer in this form. One needs to precisely define the terms _transparency_ and _explainability_ and how to measure them.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-04-18T08:36:36.133Z
Comments.Text: Thank you @nbro. I define transparency as a subset of model explainability using the taxonomy definitions in Schwalbe and Finzel 2023 https://arxiv.org/pdf/2105.07190.pdf. Specifically transparency is defined as: From a human perspective, the mechanistic understanding of a model’s decision outputs and processes by a human. Transparency is considered at the levels of simulatability (entire model), decomposability (individual components, e.g. attention parameters) and algorithmic transparency (the training algorithm).
Comments.UserDisplayName: hH1sG0n3
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-04-20T11:18:00.237Z
Comments.Text: Still, this is not mathematically precise enough. We also need to decide how to measure it. We can't simply say "yes, to me, it seems more interpretable because this notion is more familiar to me".
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-04-20T11:44:11.007Z
Comments.Text: Would you agree that in terms of decomposability, $n$-head attention would be $n$ times more complex consindering the procedure's degrees of freedom is synonymous with its complexity?
Comments.UserDisplayName: hH1sG0n3
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-04-20T14:38:04.910Z
Comments.Text: I wouldn't say that more parameters _necessarily_ implies more complexity (and thus less "explainability"), but it most likely does in many cases. I never read the paper you mentioned, but I think one approach to measure explainability may be to compare it to other obviously explainable AI techniques, like decision trees, which, at inference time, are just if-then-else statements or measure to what extent a model reasons like us (but how to measure that?). Honestly, I am currently unfamiliar with latest explainable AI literature, so maybe I can't be more helpful here.
Comments.UserDisplayName: nbro
Tags: <recurrent-neural-networks><transformer><attention><explainable-ai>
Title: In terms of explainability, is attentive RNN easier to explain than the transformer?
UserDisplayName: hH1sG0n3
ThreadText: # In terms of explainability, is attentive RNN easier to explain than the transformer?
  
  (Asked by: username_0 on 2023-04-17)
  
  Although the multi-headed attention block of the transformer allows the model to be more expressive (and therefore perform better), it is remarkably more difficult to decompose and therefore to explain (in terms of model transparency).
  
  Would an attentive RNN be considered a better architecture w.r.t. model transparency and attention-based explainability?
  
  **Tags:** recurrent-neural-networks, transformer, attention, explainable-ai
  > **Comments:**
  > **username_1**: Interesting question, but I think it's not easy to answer in this form. One needs to precisely define the terms _transparency_ and _explainability_ and how to measure them.  
  > **username_0**: Thank you @nbro. I define transparency as a subset of model explainability using the taxonomy definitions in Schwalbe and Finzel 2023 https://arxiv.org/pdf/2105.07190.pdf. Specifically transparency is defined as: From a human perspective, the mechanistic understanding of a model’s decision outputs and processes by a human. Transparency is considered at the levels of simulatability (entire model), decomposability (individual components, e.g. attention parameters) and algorithmic transparency (the training algorithm).  
  > **username_1**: Still, this is not mathematically precise enough. We also need to decide how to measure it. We can't simply say "yes, to me, it seems more interpretable because this notion is more familiar to me".  
  > **username_0**: Would you agree that in terms of decomposability, $n$-head attention would be $n$ times more complex consindering the procedure's degrees of freedom is synonymous with its complexity?  
  > **username_1**: I wouldn't say that more parameters _necessarily_ implies more complexity (and thus less "explainability"), but it most likely does in many cases. I never read the paper you mentioned, but I think one approach to measure explainability may be to compare it to other obviously explainable AI techniques, like decision trees, which, at inference time, are just if-then-else statements or measure to what extent a model reasons like us (but how to measure that?). Honestly, I am currently unfamiliar with latest explainable AI literature, so maybe I can't be more helpful here.  
  
  ---
  
  ## 0 answers
  
  