Body: <h5>Background</h5>
  <p>I am computing the attribution scores for a simple LSTM model using <a href="https://arxiv.org/pdf/1703.01365.pdf" rel="nofollow noreferrer">Integrated Gradients</a>. This method defines the contribution of a feature to a model prediction by integrating over the gradients along a path between the input and a fixed baseline:</p>
  <p><span class="math-container">$$IG_i(x) = (x_i - x'_i) \cdot\int_{\alpha=0}^1 \frac{\partial F(x'+\alpha(x-x'))}{\partial x_i}d\alpha$$</span></p>
  <p>A common way of measuring the quality of the generated attributions is via the <strong>completeness axiom</strong>, which states that:</p>
  <p><span class="math-container">$$\sum_i IG_i(x) = F(x) - F(x')$$</span></p>
  <p>The key to computing the IG scores is the approximation of the path integral, which can be approximated via Riemann sum, or a similar interpolation method. In section 5 of the IG paper, it is stated that, in practice, between 50 to 300 interpolation steps are sufficient to obtain IG scores that converge to satisfy the completeness axiom.</p>
  <h5>Issue</h5>
  <p>I am now testing the IG attributions on a simple LSTM model (1-layer, 16 hidden units). For shorter inputs (&lt;20 tokens), convergence is reached in a reasonable number of steps, and the approximation of the integral is stable. However, when the length of the input increases, I find that the integral approximation <em>diverges</em> when the number of interpolation steps is increased! This can be seen in the following plot (<em>N.B. the y axis is logarithmic</em>):</p>
  <p><a href="https://i.sstatic.net/26nh6.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/26nh6.png" alt="Delta scores for IG" /></a></p>
  <h5>Question</h5>
  <p>My question is: <em>why does the number of input tokens to the LSTM have an impact on the convergence of integrated gradients?</em></p>
  <p>It is stated in footnote 1 of the IG paper that the completeness axiom depends on whether the model satisfies <a href="https://en.wikipedia.org/wiki/Riemann_integral#Integrability" rel="nofollow noreferrer">Lebesgue's integrability condition</a>. It would surprise me, however, that increasing the number of input tokens would dissatisfy this constraint: <em>would it be possible that the model has become <em>too nonlinear</em> for numerical integration to still work? If so, are there alternative numerical integration methods that could be used here, instead of Riemann approximations or the Gauss-Legendre quadrature?</em></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-04T12:46:31.420Z
Comments.Text: Hello. Welcome to Artificial Intelligence SE. This seems to be an interesting question, even for people not familiar with IG. But can you clarify what you mean by "LSTM steps"? Do they refer to "interpolation steps"? If that's the case, what exactly do you mean by "interpolation steps" in this context?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-04T14:47:29.440Z
Comments.Text: By LSTM step I mean a single forward step of an LSTM cell: so processing a single token and up to the next. Concretely: an input string consisting of 10 tokens would amount to processing 10 LSTM steps in order to obtain the final hidden state that is used to create a model prediction.
Comments.UserDisplayName: jumelet
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-04T14:55:33.710Z
Comments.Text: By interpolation steps I mean the steps in the sense of how a Riemann sum splits up the calculation of a path integral into smaller steps. The higher this number of steps is the more accurate the approximation of the integral will become.
Comments.UserDisplayName: jumelet
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-04T15:00:42.533Z
Comments.Text: I have updated the question a little bit to make the use of "steps" less confusing. Thanks also for your edits to the question, looks nice now :)
Comments.UserDisplayName: jumelet
Tags: <neural-networks><deep-learning><long-short-term-memory><papers><convergence>
Title: Why does the number of input tokens to an LSTM have an impact on the convergence of Integrated Gradients?
UserDisplayName: jumelet
ThreadText: # Why does the number of input tokens to an LSTM have an impact on the convergence of Integrated Gradients?
  
  (Asked by: username_0 on 2021-11-03)
  
  ##### Background
  
  I am computing the attribution scores for a simple LSTM model using [Integrated Gradients](https://arxiv.org/pdf/1703.01365.pdf). This method defines the contribution of a feature to a model prediction by integrating over the gradients along a path between the input and a fixed baseline:
  
  $$IG\_i(x) = (x\_i - x'\_i) \cdot\int\_{\alpha=0}^1 \frac{\partial F(x'+\alpha(x-x'))}{\partial x\_i}d\alpha$$
  
  A common way of measuring the quality of the generated attributions is via the **completeness axiom**, which states that:
  
  $$\sum\_i IG\_i(x) = F(x) - F(x')$$
  
  The key to computing the IG scores is the approximation of the path integral, which can be approximated via Riemann sum, or a similar interpolation method. In section 5 of the IG paper, it is stated that, in practice, between 50 to 300 interpolation steps are sufficient to obtain IG scores that converge to satisfy the completeness axiom.
  
  ##### Issue
  
  I am now testing the IG attributions on a simple LSTM model (1-layer, 16 hidden units). For shorter inputs (<20 tokens), convergence is reached in a reasonable number of steps, and the approximation of the integral is stable. However, when the length of the input increases, I find that the integral approximation *diverges* when the number of interpolation steps is increased! This can be seen in the following plot (*N.B. the y axis is logarithmic*):
  
  [![Delta scores for IG](https://i.sstatic.net/26nh6.png)](https://i.sstatic.net/26nh6.png)
  
  ##### Question
  
  My question is: *why does the number of input tokens to the LSTM have an impact on the convergence of integrated gradients?*
  
  It is stated in footnote 1 of the IG paper that the completeness axiom depends on whether the model satisfies [Lebesgue's integrability condition](https://en.wikipedia.org/wiki/Riemann_integral#Integrability). It would surprise me, however, that increasing the number of input tokens would dissatisfy this constraint: *would it be possible that the model has become *too nonlinear* for numerical integration to still work? If so, are there alternative numerical integration methods that could be used here, instead of Riemann approximations or the Gauss-Legendre quadrature?*
  
  **Tags:** neural-networks, deep-learning, long-short-term-memory, papers, convergence
  > **Comments:**
  > **username_1**: Hello. Welcome to Artificial Intelligence SE. This seems to be an interesting question, even for people not familiar with IG. But can you clarify what you mean by "LSTM steps"? Do they refer to "interpolation steps"? If that's the case, what exactly do you mean by "interpolation steps" in this context?  
  > **username_0**: By LSTM step I mean a single forward step of an LSTM cell: so processing a single token and up to the next. Concretely: an input string consisting of 10 tokens would amount to processing 10 LSTM steps in order to obtain the final hidden state that is used to create a model prediction.  
  > **username_0**: By interpolation steps I mean the steps in the sense of how a Riemann sum splits up the calculation of a path integral into smaller steps. The higher this number of steps is the more accurate the approximation of the integral will become.  
  > **username_0**: I have updated the question a little bit to make the use of "steps" less confusing. Thanks also for your edits to the question, looks nice now :)  
  
  ---
  
  ## 0 answers
  
  