Answers.Body: <p>You don't need padding for attention with variable-length inputs. Looking at the formulation in the article:
  <a href="https://i.sstatic.net/u6VTP.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/u6VTP.png" alt="Attention description from article: https://medium.com/swlh/a-simple-overview-of-rnn-lstm-and-attention-mechanism-9e844763d07b" /></a></p>
  <p>A learned model (<span class="math-container">$\mathbf{a}(\cdot)$</span>) encodes the hidden state of each input token directly. This means that if you have a longer/shorter input, you would just run more forward-passes.</p>
  <p>During training, <span class="math-container">$\mathbf{a}(\cdot)$</span> learns to give higher values for more relevant tokens. It only depends on the hidden-dimensions of <span class="math-container">$h_j$</span> and <span class="math-container">$s_{t-1}$</span>. You might still use masking and padding, this generally is more of a efficiency thing (because GPUs like static data sizes). In fact, you'll probably have to use masking to make sure that your decoder doesn't attend to masking tokens.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-06T07:18:41.130Z
Answers.Text: Oh I see, so we just do as many forward passes as potential hidden states we want to attend to, each forward pass giving a score and then we apply a softmax!
Answers.UserDisplayName: FluidMechanics Potential Flows
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-06T18:07:37.953Z
Answers.Text: Yep, that's correct
Answers.UserDisplayName: Alexander Wan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-06T01:43:16.423Z
Answers.LastActivityDate: 2024-02-06T01:43:16.423Z
Answers.UserDisplayName: Alexander Wan
Answers.Body: <p>Two concepts are critical here: masking and padding.</p>
  <p>From <a href="https://www.tensorflow.org/guide/keras/understanding_masking_and_padding" rel="nofollow noreferrer">Tensorflow</a>:</p>
  <blockquote>
  <p><strong>Masking</strong> is a way to tell sequence-processing layers that certain
  timesteps in an input are missing, and thus should be skipped when
  processing the data.</p>
  <p><strong>Padding</strong> is a special form of masking where the masked steps are at the
  start or the end of a sequence. Padding comes from the need to encode
  sequence data into contiguous batches: in order to make all sequences
  in a batch fit a given standard length, it is necessary to pad or
  truncate some sequences.</p>
  </blockquote>
  <p>See the article for more details.</p>
  <p>Also see <a href="https://pyimagesearch.com/2022/07/25/introduction-to-recurrent-neural-networks-with-keras-and-tensorflow/" rel="nofollow noreferrer">Introduction to Recurrent Neural Networks with Keras and TensorFlow by Aritra Roy Gosthipaty, Devjyoti Chakraborty, and Ritwik Raha</a>. See the section entitled <a href="https://pyimagesearch.com/2022/07/25/introduction-to-recurrent-neural-networks-with-keras-and-tensorflow/#h2Caveat" rel="nofollow noreferrer">&quot;A Caveat: Masking and Padding&quot;</a>.</p>
  <p>Prudhviraju Srivatsavaya explains the concepts in reference to Transformers in his article <a href="https://medium.com/@prudhviraju.srivatsavaya/attention-masks-explanation-b4ee2e8f8d23" rel="nofollow noreferrer">Attention Masks - Explanation</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-05T07:32:39.440Z
Answers.Text: So does that mean attention has a set length and if we have let's say more input tokens that its set length it would not be able to provide attention to *any* previous hidden states?
Answers.UserDisplayName: FluidMechanics Potential Flows
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-04T21:01:36.377Z
Answers.LastActivityDate: 2024-02-04T21:07:23.517Z
Answers.LastEditDate: 2024-02-04T21:07:23.517Z
Answers.UserDisplayName: Brian O'Donnell
Body: <p>In RNNs, to avoid &quot;forgetting&quot; information encoded by earlier encoders, we can use attention. It's basically a second neural network that tells us how much we should attend at time <em>t</em> on each of the earlier hidden states (from <em>1</em> to <em>t - 1</em>).
  This is described here:
  <a href="https://i.sstatic.net/cri6k.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/cri6k.png" alt="enter image description here" /></a>
  (image from <a href="https://medium.com/swlh/a-simple-overview-of-rnn-lstm-and-attention-mechanism-9e844763d07b" rel="nofollow noreferrer">https://medium.com/swlh/a-simple-overview-of-rnn-lstm-and-attention-mechanism-9e844763d07b</a>)</p>
  <p>However, there is something I don't understand. This second neural network will output <em>t</em> values. So there will be <em>t</em> neurons on the last layer. What if my input sentence (in a seq2seq network for example) has only <em>t - 2</em> words. Or <em>t + 2</em> words. Then I can't use that same second neural network since its architecture is fixed. How is it dealt?</p>
  <p>EDIT: is it that the second neural network takes as input one encoder hidden state and one decoder hidden state, and we get <em>t</em> scores by doing <em>t</em> forward passes (one per each input hidden state, all with the current decoder hidden state)? And then we softmax all those <em>t</em> outputs?</p>
  
Tags: <recurrent-neural-networks><attention>
Title: Attention with Recurrent Neural Networks
UserDisplayName: FluidMechanics Potential Flows
ThreadText: # Attention with Recurrent Neural Networks
  
  (Asked by: username_0 on 2024-02-02)
  
  In RNNs, to avoid "forgetting" information encoded by earlier encoders, we can use attention. It's basically a second neural network that tells us how much we should attend at time *t* on each of the earlier hidden states (from *1* to *t - 1*).
  This is described here:
  [![enter image description here](https://i.sstatic.net/cri6k.png)](https://i.sstatic.net/cri6k.png)
  (image from <https://medium.com/swlh/a-simple-overview-of-rnn-lstm-and-attention-mechanism-9e844763d07b>)
  
  However, there is something I don't understand. This second neural network will output *t* values. So there will be *t* neurons on the last layer. What if my input sentence (in a seq2seq network for example) has only *t - 2* words. Or *t + 2* words. Then I can't use that same second neural network since its architecture is fixed. How is it dealt?
  
  EDIT: is it that the second neural network takes as input one encoder hidden state and one decoder hidden state, and we get *t* scores by doing *t* forward passes (one per each input hidden state, all with the current decoder hidden state)? And then we softmax all those *t* outputs?
  
  **Tags:** recurrent-neural-networks, attention
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2024-02-06. Score: 1 | Accepted answer
  
  You don't need padding for attention with variable-length inputs. Looking at the formulation in the article:
  [![Attention description from article: https://medium.com/swlh/a-simple-overview-of-rnn-lstm-and-attention-mechanism-9e844763d07b](https://i.sstatic.net/u6VTP.png)](https://i.sstatic.net/u6VTP.png)
  
  A learned model ($\mathbf{a}(\cdot)$) encodes the hidden state of each input token directly. This means that if you have a longer/shorter input, you would just run more forward-passes.
  
  During training, $\mathbf{a}(\cdot)$ learns to give higher values for more relevant tokens. It only depends on the hidden-dimensions of $h\_j$ and $s\_{t-1}$. You might still use masking and padding, this generally is more of a efficiency thing (because GPUs like static data sizes). In fact, you'll probably have to use masking to make sure that your decoder doesn't attend to masking tokens.
  
  > **Comments:**
  > **username_0**: Oh I see, so we just do as many forward passes as potential hidden states we want to attend to, each forward pass giving a score and then we apply a softmax!  
  > **username_1**: Yep, that's correct  
  
  ---
  
  ### Answer by username_2 on 2024-02-04. Score: 0
  
  Two concepts are critical here: masking and padding.
  
  From [Tensorflow](https://www.tensorflow.org/guide/keras/understanding_masking_and_padding):
  
  > **Masking** is a way to tell sequence-processing layers that certain
  > timesteps in an input are missing, and thus should be skipped when
  > processing the data.
  >
  > **Padding** is a special form of masking where the masked steps are at the
  > start or the end of a sequence. Padding comes from the need to encode
  > sequence data into contiguous batches: in order to make all sequences
  > in a batch fit a given standard length, it is necessary to pad or
  > truncate some sequences.
  
  See the article for more details.
  
  Also see [Introduction to Recurrent Neural Networks with Keras and TensorFlow by Aritra Roy Gosthipaty, Devjyoti Chakraborty, and Ritwik Raha](https://pyimagesearch.com/2022/07/25/introduction-to-recurrent-neural-networks-with-keras-and-tensorflow/). See the section entitled ["A Caveat: Masking and Padding"](https://pyimagesearch.com/2022/07/25/introduction-to-recurrent-neural-networks-with-keras-and-tensorflow/#h2Caveat).
  
  Prudhviraju Srivatsavaya explains the concepts in reference to Transformers in his article [Attention Masks - Explanation](https://medium.com/@prudhviraju.srivatsavaya/attention-masks-explanation-b4ee2e8f8d23).
  
  > **Comments:**
  > **username_0**: So does that mean attention has a set length and if we have let's say more input tokens that its set length it would not be able to provide attention to *any* previous hidden states?  
  
  ---
  
  