Body: <p>Suppose that we are doing machine translation. We have a conditional language model  with attention where we are are trying to predict a sequence <span class="math-container">$y_1, y_2, \dots, y_J$</span> from <span class="math-container">$x_1, x_2, \dots x_I$</span>: <span class="math-container">$$P(y_1, y_2, \dots, y_{J}|x_1, x_2, \dots x_I) = \prod_{j=1}^{J} p(y_j|v_j, y_1, \dots, y_{j-1})$$</span> where <span class="math-container">$v_j$</span> is a context vector that is different for each <span class="math-container">$y_j$</span>. Using an RNN with a encoder-decoder structure, each element <span class="math-container">$x_i$</span> of the input sequence and <span class="math-container">$y_j$</span> of the output sequence is converted into an embedding <span class="math-container">$h_i$</span> and <span class="math-container">$s_j$</span> respectively: <span class="math-container">$$h_i = f(h_{i-1}, x_i) \\ s_j = g(s_{j-1},[y_{j-1}, v_j])$$</span> where <span class="math-container">$f$</span> is some function of the previous input state <span class="math-container">$h_{i-1}$</span> and the current input word <span class="math-container">$x_i$</span> and <span class="math-container">$g$</span> is some function of the previous output state <span class="math-container">$s_{j-1}$</span>, the previous output word <span class="math-container">$y_{j-1}$</span> and the context vector <span class="math-container">$v_j$</span>.</p>
  
  <p>Now, we want the process of predicting <span class="math-container">$s_j$</span> to "pay attention" to the correct parts of the encoder states (context vector <span class="math-container">$v_j$</span>). So: <span class="math-container">$$v_j = \sum_{i=1}^{I} \alpha_{ij} h_i$$</span> where <span class="math-container">$\alpha_{ij}$</span> tells us how much weight to put on the <span class="math-container">$i^{th}$</span> state of the source vector when predicting the <span class="math-container">$j^{th}$</span> word of the output vector. Since we want the <span class="math-container">$\alpha_{ij}$</span>s to be probabilities, we use a softmax function on the similarities between the encoder and decoder states: <span class="math-container">$$\alpha_{ij} = \frac{\exp(\text{sim}(h_i, s_{j-1}))}{\sum_{i'=1}^{I} \exp(\text{sim}(h_i, s_{j-1}))}$$</span></p>
  
  <p>Now, in additive attention, the similarities of the encoder and decoder states are computed as: <span class="math-container">$$\text{sim}(h_i, s_{j}) = \textbf{w}^{T} \text{tanh}(\textbf{W}_{h}h_{i} +\textbf{W}_{s}s_{j})$$</span></p>
  
  <p>where <span class="math-container">$\textbf{w}$</span>, <span class="math-container">$\textbf{W}_{h}$</span> and <span class="math-container">$\textbf{W}_{s}$</span> are learned attention parameters using a  one-hidden layer feed-forward network. </p>
  
  <p>What is the intuition behind this definition? Why use the <span class="math-container">$\text{tanh}$</span> function? I know that the idea is to use one layer of a neural network to predict the similarities. </p>
  
  <p><strong>Added.</strong> This description of machine translation/attention is based on the Coursera course <a href="https://www.coursera.org/learn/language-processing/lecture/1nQaG/attention-mechanism" rel="nofollow noreferrer">Natural Language Processing</a>.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-02-09T01:25:45.327Z
Comments.Text: I would first like to note that $\tanh$ produces numbers in the range $[-1, 1]$. So, you are doing a dot product between $w^T$ and a vector of numbers in the range $[-1, 1]$. Recall also that if you multiply any number $x$ by another number $y$ between $0$ and $1$, the result is a smaller number than $x$.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-09-13T07:41:48.300Z
Comments.Text: This is a very good question. I am surprised no one answered it.
Comments.UserDisplayName: Long
Tags: <recurrent-neural-networks><attention><machine-translation>
Title: What is the intuition behind the calculation of the similarity between encoder and decoder states?
UserDisplayName: AIguy122
ThreadText: # What is the intuition behind the calculation of the similarity between encoder and decoder states?
  
  (Asked by: username_0 on 2019-02-08)
  
  Suppose that we are doing machine translation. We have a conditional language model with attention where we are are trying to predict a sequence $y\_1, y\_2, \dots, y\_J$ from $x\_1, x\_2, \dots x\_I$: $$P(y\_1, y\_2, \dots, y\_{J}|x\_1, x\_2, \dots x\_I) = \prod\_{j=1}^{J} p(y\_j|v\_j, y\_1, \dots, y\_{j-1})$$ where $v\_j$ is a context vector that is different for each $y\_j$. Using an RNN with a encoder-decoder structure, each element $x\_i$ of the input sequence and $y\_j$ of the output sequence is converted into an embedding $h\_i$ and $s\_j$ respectively: $$h\_i = f(h\_{i-1}, x\_i) \\ s\_j = g(s\_{j-1},[y\_{j-1}, v\_j])$$ where $f$ is some function of the previous input state $h\_{i-1}$ and the current input word $x\_i$ and $g$ is some function of the previous output state $s\_{j-1}$, the previous output word $y\_{j-1}$ and the context vector $v\_j$.
  
  Now, we want the process of predicting $s\_j$ to "pay attention" to the correct parts of the encoder states (context vector $v\_j$). So: $$v\_j = \sum\_{i=1}^{I} \alpha\_{ij} h\_i$$ where $\alpha\_{ij}$ tells us how much weight to put on the $i^{th}$ state of the source vector when predicting the $j^{th}$ word of the output vector. Since we want the $\alpha\_{ij}$s to be probabilities, we use a softmax function on the similarities between the encoder and decoder states: $$\alpha\_{ij} = \frac{\exp(\text{sim}(h\_i, s\_{j-1}))}{\sum\_{i'=1}^{I} \exp(\text{sim}(h\_i, s\_{j-1}))}$$
  
  Now, in additive attention, the similarities of the encoder and decoder states are computed as: $$\text{sim}(h\_i, s\_{j}) = \textbf{w}^{T} \text{tanh}(\textbf{W}\_{h}h\_{i} +\textbf{W}\_{s}s\_{j})$$
  
  where $\textbf{w}$, $\textbf{W}\_{h}$ and $\textbf{W}\_{s}$ are learned attention parameters using a one-hidden layer feed-forward network.
  
  What is the intuition behind this definition? Why use the $\text{tanh}$ function? I know that the idea is to use one layer of a neural network to predict the similarities.
  
  **Added.** This description of machine translation/attention is based on the Coursera course [Natural Language Processing](https://www.coursera.org/learn/language-processing/lecture/1nQaG/attention-mechanism).
  
  **Tags:** recurrent-neural-networks, attention, machine-translation
  
  > **Comments:**
  > **username_1**: I would first like to note that $\tanh$ produces numbers in the range $[-1, 1]$. So, you are doing a dot product between $w^T$ and a vector of numbers in the range $[-1, 1]$. Recall also that if you multiply any number $x$ by another number $y$ between $0$ and $1$, the result is a smaller number than $x$.  
  > **username_2**: This is a very good question. I am surprised no one answered it.  
  
  ---
  
  ## 0 answers
  
  