Answers.Body: <p>Why does it look dumb? If you think the model makes sense, then the explanation seems reasonable to me.</p>
  <p>I would not call this gradient-based explanation, however, since the features in this case are discrete, not continuous (you cannot take a   derivative wrt to <span class="math-container">$x_1$</span> and <span class="math-container">$x_2$</span>, if those are discrete features)</p>
  <p>You can however look at ablations (i.e. change in scores if the word is not present), which would be the analogous of gradients for discrete features. The interpretation is: if that word is present, the score increases by this much. Why has this text been classified as toxic? Because it contains the following toxic words, each sorted by their &quot;toxicity&quot;.</p>
  <p>Of course that would be a very simple model; words do not occur in isolation and looking at their mere presence completely disregard the context in which they appear. However, that's a problem of the model, not of the explanation of the model.</p>
  <p>The more sophisticated models these days use as base architecture the transformer, which is able to differentiate words based on the context in which they occur. There are some explanatory models for transformer-based architectures, but they will not simply assign a numerical score to a word (as that cannot accurately reflect the predictions of the model, since it ignores context).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-04-07T10:59:50.793Z
Answers.Text: Thanks for your comment, Ant! However I don't get how "gradient-friendly" the ablation model would be. The change in scores, if the word is not present, would be very "binary" and therefore not very differentiable. Unless you "ablate" word after word and look at the change in the score? I have another question, do you know which NLP methods use gradient and activation methods?
Answers.UserDisplayName: Revolucion for Monica
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-04-05T20:17:16.590Z
Answers.LastActivityDate: 2022-04-05T20:17:16.590Z
Answers.UserDisplayName: Ant
Body: <p>I am looking for a simple example of gradient-based methods for NLP. More specifically I am looking for post-hoc local explanations gradient-based methods, that is to say, which explain a single prediction by performing additional operations (after the model has emitted a prediction). Here is an example of a gradient-based method for &quot;Finance&quot;? (I'm not very sure of what domain it is as it is not NLP, nor Computer Vision, nor time series ...):</p>
  <blockquote>
  <p>Let us consider a minimal example, where a linear regression is used to estimate the future capital asset <span class="math-container">$y_{c}$</span>, based on two investments <span class="math-container">$x_{1}$</span> and <span class="math-container">$x_{2}$</span>. Let assume the assumptions above are met and the model parameters are estimated as follows:
  <span class="math-container">$$\mathbb{E}\left[y_{c} \mid x_{1}, x_{2}\right]=1.05 x_{1}+1.50 x_{2},$$</span>
  We can derive immediately a global interpretation of this model. Every dollar invested in fund <span class="math-container">$x_{1}$</span> will produce capital of <span class="math-container">$1.05 \\\$$</span>, while every dollar invested in <span class="math-container">$x_{2}$</span> will produce a capital of <span class="math-container">$1.50 \\\$$</span>, independently of the values <span class="math-container">$x_{1}$</span> and <span class="math-container">$x_{2}$</span> might assume in a concrete scenario. Notice that this explanation is purely based on the learned coefficient <span class="math-container">$w_{1}=1.05$</span> and <span class="math-container">$w_{2}=1.50$</span>. These, sometimes called partial regression coefficients, are themselves candidate attribution values to explain the influence of the independent variables of the target variable:
  <span class="math-container">$$R_{1}(x)=1.05 \quad R_{2}(x)=1.50$$</span>
  Notice also that the coefficients are the partial derivatives of the target variable with respect to the independent variable, therefore this attribution is nothing but the model gradient.</p>
  </blockquote>
  <p>I thought about transformating it to an NLP example like:</p>
  <p><em>Let us consider a minimal example, where a linear regression is used to estimate the toxicity of a text <span class="math-container">$y_{c}$</span>, based on the occurence of words <span class="math-container">$x_{1}$</span> and <span class="math-container">$x_{2}$</span>. Let's assume the assumptions above are met and the model parameters are estimated as follows:</em></p>
  <p><span class="math-container">$$\mathbb{E}\left[y_{c} \mid x_{1}, x_{2}\right]=1.05 x_{1}+1.50 x_{2},$$</span></p>
  <p><em>We can derive immediately a global interpretation of this model. Every occurrence of word <span class="math-container">$x_{1}$</span> will produce a toxicity of <span class="math-container">$1.05$</span>, while every occurrence of word in <span class="math-container">$x_{2}$</span> will produce a capital of <span class="math-container">$1.50$</span>, independently of the values <span class="math-container">$x_{1}$</span> and <span class="math-container">$x_{2}$</span> might assume in a concrete scenario. Notice that this explanation is purely based on the learned coefficient <span class="math-container">$w_{1}=1.05$</span> and <span class="math-container">$w_{2}=1.50$</span>. These, sometimes called partial regression coefficients, are themselves candidate attribution values to explain the influence of the independent variables of the target variable</em></p>
  <p>That look pretty dumb, isn't it? Do you have any better example</p>
  <h1>Update : does the ablation model use gradient?</h1>
  <p>So it seems my model isn't so dumb, but Ant gave the idea of using the ablation model. I didn't know about this technique but it seems straightforward: you iterate over your text and get rid of a given word in part of the text and see how much the text gains or loses toxicity. It looked very gradient-friendly.</p>
  <p>However, I tried to create an ablation algorithm that looks at the k next words and tries to get the toxicity of the word by doing the difference between the text predicted toxicity <span class="math-container">$\hat y$</span> and the same one when the text is being ablated <span class="math-container">$\hat y_{ablated}$</span>. I inspired myself from <a href="https://www.mdpi.com/2076-3417/12/3/1427/pdf" rel="nofollow noreferrer">Warren Freeborough's example for time series in his paper page 5</a>.</p>
  <p><a href="https://i.sstatic.net/vAswe.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/vAswe.png" alt="enter image description here" /></a></p>
  <p>I get an en error, or sensibility, matrix <span class="math-container">$E_{avg}$</span>. But I don't see where any gradient has been used. At most I could say that the model can use gradient. For instance, if it is an RNN, it uses a gradient method, isn't it? But I don't see where ablation uses gradient.</p>
  
Tags: <natural-language-processing>
Title: What are examples of simple gradient based NLP models?
UserDisplayName: Revolucion for Monica
ThreadText: # What are examples of simple gradient based NLP models?
  
  (Asked by: username_0 on 2022-04-05)
  
  I am looking for a simple example of gradient-based methods for NLP. More specifically I am looking for post-hoc local explanations gradient-based methods, that is to say, which explain a single prediction by performing additional operations (after the model has emitted a prediction). Here is an example of a gradient-based method for "Finance"? (I'm not very sure of what domain it is as it is not NLP, nor Computer Vision, nor time series ...):
  
  > Let us consider a minimal example, where a linear regression is used to estimate the future capital asset $y\_{c}$, based on two investments $x\_{1}$ and $x\_{2}$. Let assume the assumptions above are met and the model parameters are estimated as follows:
  > $$\mathbb{E}\left[y\_{c} \mid x\_{1}, x\_{2}\right]=1.05 x\_{1}+1.50 x\_{2},$$
  > We can derive immediately a global interpretation of this model. Every dollar invested in fund $x\_{1}$ will produce capital of $1.05 \\\$$, while every dollar invested in $x\_{2}$ will produce a capital of $1.50 \\\$$, independently of the values $x\_{1}$ and $x\_{2}$ might assume in a concrete scenario. Notice that this explanation is purely based on the learned coefficient $w\_{1}=1.05$ and $w\_{2}=1.50$. These, sometimes called partial regression coefficients, are themselves candidate attribution values to explain the influence of the independent variables of the target variable:
  > $$R\_{1}(x)=1.05 \quad R\_{2}(x)=1.50$$
  > Notice also that the coefficients are the partial derivatives of the target variable with respect to the independent variable, therefore this attribution is nothing but the model gradient.
  
  I thought about transformating it to an NLP example like:
  
  *Let us consider a minimal example, where a linear regression is used to estimate the toxicity of a text $y\_{c}$, based on the occurence of words $x\_{1}$ and $x\_{2}$. Let's assume the assumptions above are met and the model parameters are estimated as follows:*
  
  $$\mathbb{E}\left[y\_{c} \mid x\_{1}, x\_{2}\right]=1.05 x\_{1}+1.50 x\_{2},$$
  
  *We can derive immediately a global interpretation of this model. Every occurrence of word $x\_{1}$ will produce a toxicity of $1.05$, while every occurrence of word in $x\_{2}$ will produce a capital of $1.50$, independently of the values $x\_{1}$ and $x\_{2}$ might assume in a concrete scenario. Notice that this explanation is purely based on the learned coefficient $w\_{1}=1.05$ and $w\_{2}=1.50$. These, sometimes called partial regression coefficients, are themselves candidate attribution values to explain the influence of the independent variables of the target variable*
  
  That look pretty dumb, isn't it? Do you have any better example
  
  # Update : does the ablation model use gradient?
  
  So it seems my model isn't so dumb, but Ant gave the idea of using the ablation model. I didn't know about this technique but it seems straightforward: you iterate over your text and get rid of a given word in part of the text and see how much the text gains or loses toxicity. It looked very gradient-friendly.
  
  However, I tried to create an ablation algorithm that looks at the k next words and tries to get the toxicity of the word by doing the difference between the text predicted toxicity $\hat y$ and the same one when the text is being ablated $\hat y\_{ablated}$. I inspired myself from [Warren Freeborough's example for time series in his paper page 5](https://www.mdpi.com/2076-3417/12/3/1427/pdf).
  
  [![enter image description here](https://i.sstatic.net/vAswe.png)](https://i.sstatic.net/vAswe.png)
  
  I get an en error, or sensibility, matrix $E\_{avg}$. But I don't see where any gradient has been used. At most I could say that the model can use gradient. For instance, if it is an RNN, it uses a gradient method, isn't it? But I don't see where ablation uses gradient.
  
  **Tags:** natural-language-processing
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-04-05. Score: 0
  
  Why does it look dumb? If you think the model makes sense, then the explanation seems reasonable to me.
  
  I would not call this gradient-based explanation, however, since the features in this case are discrete, not continuous (you cannot take a derivative wrt to $x\_1$ and $x\_2$, if those are discrete features)
  
  You can however look at ablations (i.e. change in scores if the word is not present), which would be the analogous of gradients for discrete features. The interpretation is: if that word is present, the score increases by this much. Why has this text been classified as toxic? Because it contains the following toxic words, each sorted by their "toxicity".
  
  Of course that would be a very simple model; words do not occur in isolation and looking at their mere presence completely disregard the context in which they appear. However, that's a problem of the model, not of the explanation of the model.
  
  The more sophisticated models these days use as base architecture the transformer, which is able to differentiate words based on the context in which they occur. There are some explanatory models for transformer-based architectures, but they will not simply assign a numerical score to a word (as that cannot accurately reflect the predictions of the model, since it ignores context).
  
  > **Comments:**
  > **username_0**: Thanks for your comment, Ant! However I don't get how "gradient-friendly" the ablation model would be. The change in scores, if the word is not present, would be very "binary" and therefore not very differentiable. Unless you "ablate" word after word and look at the change in the score? I have another question, do you know which NLP methods use gradient and activation methods?  
  
  ---
  
  