Answers.Body: <blockquote>
  <p>Evidently you can only initialize it (<span class="math-container">$\vec{c_t}$</span>) once</p>
  </blockquote>
  <p>As I see it, <span class="math-container">$\vec{c_t}$</span> depends on <span class="math-container">$\vec{h}[1] \ldots \vec{h}[n]$</span> <strong>AND</strong> <span class="math-container">$\vec{s_{t-1}}$</span> (because <span class="math-container">$\alpha_i[t]$</span> depend on <span class="math-container">$\vec{s_{t-1}}$</span>, and so is different for every calculation of a new <span class="math-container">$\vec{s_t}$</span>).</p>
  <p>And so <span class="math-container">$\vec{c_t}$</span> is different for every <span class="math-container">$\vec{s_t}$</span>.</p>
  <p>It can be used by the decoder by e.g., concatenating it with <span class="math-container">$\vec{s_{t-1}}$</span>, or with <span class="math-container">$\vec{y_{t-1}}$</span>, or by adding instead of concatenating, or ...</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-24T10:46:53.770Z
Answers.Text: Your clarification of `it` is incorrect. `it` in that sentence stands for $\vec s$, the decoder hidden state. See the preceding sentence: *(...) the decoder as initialisation for its hidden state. Evidently, you can only initialise it once.* I also indicated that $\alpha_i[t]$ is time-dependent, both with the $[t]$ parameter and by stating *(...) the decoder computes weights $\alpha_i[t]$ with each iteration -- by comparing the decoder's previous hidden state $\vec s[t−1]$ (...)*.
  
  Do you have a source for your last paragraph? It's the usage of $\vec c[t]$ I'm looking for (hence the title).
Answers.UserDisplayName: Mew
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-24T11:01:33.000Z
Answers.Text: Right, I misinterpreted what you meant by 'it'.
  It seems to me you can only initialize $\vec{s_0}$ once, sure, but for the calculation of each subsequent $\vec{s_t}$, a different context vector $\vec{c_t}$ is used, combining the (fixed) encoder-hidden-states $\vec{h}[1] \ldots \vec{h}[n] $ with different $\alpha$ weights.
Answers.UserDisplayName: Nathan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-24T11:03:38.437Z
Answers.Text: For the last paragraph, this is just to illustrate how it could be done. However, it seems that the appendix in the Bahdanau papers shows the details of how $\vec{c_t}$  is used:  https://arxiv.org/pdf/1409.0473.pdf#page=12, apparently roughly by adding (after projection) it to (projected versions of) $\vec{s_{i-1}}$ and $\vec{y_{i-1}}$ .
Answers.UserDisplayName: Nathan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-24T10:20:57.190Z
Answers.LastActivityDate: 2022-10-24T10:20:57.190Z
Answers.UserDisplayName: Nathan
Answers.Body: <p>If our similarity function is defined as</p>
  <p><span class="math-container">$$e^{t,t'}= f(y^t,h^{t'})$$</span>
  for a <span class="math-container">$(t, t')$</span> pair, this would give you an attention weight/score for each hidden state <span class="math-container">$h^{t'}$</span>. In this way you end up with <span class="math-container">$[0, ...,  t-1]$</span> <em>weighted hidden states</em>, which you combine using a sum. This weighted sum is the &quot;context vector&quot;
  <span class="math-container">$$ c_t = \sum_{i=0}^{T}a_{t,i}h_i$$</span></p>
  <p>To answer your question, for each position you end up with a context vector <span class="math-container">$c_t$</span> which fully replaces the hidden state <span class="math-container">$h^{t}$</span> at the exact same position of your computation graph.</p>
  <p><span class="math-container">$$ h_t \rightarrow c_t,\ as\ \ LSTM_{simple}\ \rightarrow\ LSTM_{attentive} $$</span></p>
  <p>Hope it helps!</p>
  <p><strong>References</strong>: <br />
  - Neural Machine Translation by Jointly Learning to Align and Translate, 2014 \</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-08T19:23:50.987Z
Answers.Text: This seems unlikely to me. What is this $h_t$ at the end of your post? It isn't an encoder state, so you must mean a decoder state $s_t$. Yet, Bahdanau on page 3 writes that $s_i = f(s_{i−1}, y_{i−1}, c_i)$, which suggests to me that $s_{t-1} \neq c_{t-1}$, otherwise Bahdanau would have written $s_i = f(c_{i−1}, y_{i−1}, c_i)$.
Answers.UserDisplayName: Mew
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-12T12:34:39.763Z
Answers.Text: Let me take a step back. In simple LSTM, can you define the difference between $h_t$ and $y_t$, so that i understand your pov better.
Answers.UserDisplayName: hH1sG0n3
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-12T17:44:34.907Z
Answers.Text: It doesn't matter how I define these things. That equation is a literal quote from the Bahdanau paper linked in my question. It also doesn't matter whether it's with or without attention, because the Seq2Seq decoder LSTM feeds its own output $y_t$ back to its input regardless. $h_t$ are the encoder's hidden states. $s_t$ are the decoder's hidden states.
Answers.UserDisplayName: Mew
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-08T15:36:31.970Z
Answers.LastActivityDate: 2022-10-08T15:36:31.970Z
Answers.UserDisplayName: hH1sG0n3
Body: <p>For neural machine translation, there's this model &quot;Seq2Seq with attention&quot;, also known as the &quot;<a href="https://arxiv.org/pdf/1409.0473.pdf" rel="nofollow noreferrer">Bahdanau</a> architecture&quot; (a good image can be found on <a href="https://lena-voita.github.io/nlp_course/seq2seq_and_attention.html" rel="nofollow noreferrer">this page</a>), where instead of Seq2Seq's encoder LSTM passing a single hidden vector <span class="math-container">$\vec h[T]$</span> to the decoder LSTM, the encoder makes all of its hidden vectors <span class="math-container">$\vec h[1] \dots \vec h[T]$</span> available and the decoder computes weights <span class="math-container">$\alpha_i[t]$</span> with each iteration -- by comparing the decoder's previous hidden state <span class="math-container">$\vec s[t-1]$</span> to each encoder hidden state <span class="math-container">$\vec h[i]$</span> -- to decide which of those hidden vectors are the most valuable. These are then added together to get a single &quot;context vector&quot; <span class="math-container">$\vec c[t] = \alpha_1[t]\,\vec h[1] + \alpha_2[t]\,\vec h[2]+\dots +\alpha_T[t]\,\vec h[T]$</span>, which supposedly functions as Seq2Seq's single hidden vector.</p>
  <p>But the latter can't be the case. Seq2Seq originally passed that vector to the decoder as initialisation for its hidden state. Evidently, you can only initialise it once. So then, <em>how is <span class="math-container">$\vec c[t]$</span> used by the decoder</em>? None of the sources I have read (see e.g. the original paper linked above, or <a href="https://towardsdatascience.com/sequence-2-sequence-model-with-attention-mechanism-9e9ca2a613a" rel="nofollow noreferrer">this article</a>, or <a href="https://www.researchgate.net/publication/353217198_Sequence_to_Point_Learning_Based_on_an_Attention_Neural_Network_for_Nonintrusive_Load_Decomposition" rel="nofollow noreferrer">this paper</a>, or <a href="https://lena-voita.github.io/nlp_course/seq2seq_and_attention.html" rel="nofollow noreferrer">this otherwise excellent reader</a>) disclose what happens. At most, they hide this mechanism behind &quot;a function <span class="math-container">$f$</span>&quot; which is never explained. I must be overlooking something super obvious here, apparently.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-10-08T12:13:14.863Z
Comments.Text: Hello, can you check this post I just created and lmk which part of it you would like to further break down to get a clear answer? https://datascience.stackexchange.com/a/115015/102852
Comments.UserDisplayName: hH1sG0n3
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-10-08T13:21:05.497Z
Comments.Text: @hH1sG0n3 Like all of the other sources I linked to, you calculate the attention weights and the context vector, but you never show how to **use** the context vector, only how to **get** it.
Comments.UserDisplayName: Mew
Tags: <long-short-term-memory><attention><seq2seq>
Title: How does Seq2Seq with attention actually use the attention (i.e. the context vector)?
UserDisplayName: Mew
ThreadText: # How does Seq2Seq with attention actually use the attention (i.e. the context vector)?
  
  (Asked by: username_0 on 2022-10-07)
  
  For neural machine translation, there's this model "Seq2Seq with attention", also known as the "[Bahdanau](https://arxiv.org/pdf/1409.0473.pdf) architecture" (a good image can be found on [this page](https://lena-voita.github.io/nlp_course/seq2seq_and_attention.html)), where instead of Seq2Seq's encoder LSTM passing a single hidden vector $\vec h[T]$ to the decoder LSTM, the encoder makes all of its hidden vectors $\vec h[1] \dots \vec h[T]$ available and the decoder computes weights $\alpha\_i[t]$ with each iteration -- by comparing the decoder's previous hidden state $\vec s[t-1]$ to each encoder hidden state $\vec h[i]$ -- to decide which of those hidden vectors are the most valuable. These are then added together to get a single "context vector" $\vec c[t] = \alpha\_1[t]\,\vec h[1] + \alpha\_2[t]\,\vec h[2]+\dots +\alpha\_T[t]\,\vec h[T]$, which supposedly functions as Seq2Seq's single hidden vector.
  
  But the latter can't be the case. Seq2Seq originally passed that vector to the decoder as initialisation for its hidden state. Evidently, you can only initialise it once. So then, *how is $\vec c[t]$ used by the decoder*? None of the sources I have read (see e.g. the original paper linked above, or [this article](https://towardsdatascience.com/sequence-2-sequence-model-with-attention-mechanism-9e9ca2a613a), or [this paper](https://www.researchgate.net/publication/353217198_Sequence_to_Point_Learning_Based_on_an_Attention_Neural_Network_for_Nonintrusive_Load_Decomposition), or [this otherwise excellent reader](https://lena-voita.github.io/nlp_course/seq2seq_and_attention.html)) disclose what happens. At most, they hide this mechanism behind "a function $f$" which is never explained. I must be overlooking something super obvious here, apparently.
  
  **Tags:** long-short-term-memory, attention, seq2seq
  > **Comments:**
  > **username_1**: Hello, can you check this post I just created and lmk which part of it you would like to further break down to get a clear answer? https://datascience.stackexchange.com/a/115015/102852  
  > **username_0**: @hH1sG0n3 Like all of the other sources I linked to, you calculate the attention weights and the context vector, but you never show how to **use** the context vector, only how to **get** it.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2022-10-24. Score: 1
  
  > Evidently you can only initialize it ($\vec{c\_t}$) once
  
  As I see it, $\vec{c\_t}$ depends on $\vec{h}[1] \ldots \vec{h}[n]$ **AND** $\vec{s\_{t-1}}$ (because $\alpha\_i[t]$ depend on $\vec{s\_{t-1}}$, and so is different for every calculation of a new $\vec{s\_t}$).
  
  And so $\vec{c\_t}$ is different for every $\vec{s\_t}$.
  
  It can be used by the decoder by e.g., concatenating it with $\vec{s\_{t-1}}$, or with $\vec{y\_{t-1}}$, or by adding instead of concatenating, or ...
  
  > **Comments:**
  > **username_0**: Your clarification of `it` is incorrect. `it` in that sentence stands for $\vec s$, the decoder hidden state. See the preceding sentence: *(...) the decoder as initialisation for its hidden state. Evidently, you can only initialise it once.* I also indicated that $\alpha_i[t]$ is time-dependent, both with the $[t]$ parameter and by stating *(...) the decoder computes weights $\alpha_i[t]$ with each iteration -- by comparing the decoder's previous hidden state $\vec s[t−1]$ (...)*.  Do you have a source for your last paragraph? It's the usage of $\vec c[t]$ I'm looking for (hence the title).  
  > **username_2**: Right, I misinterpreted what you meant by 'it'. It seems to me you can only initialize $\vec{s_0}$ once, sure, but for the calculation of each subsequent $\vec{s_t}$, a different context vector $\vec{c_t}$ is used, combining the (fixed) encoder-hidden-states $\vec{h}[1] \ldots \vec{h}[n] $ with different $\alpha$ weights.  
  > **username_2**: For the last paragraph, this is just to illustrate how it could be done. However, it seems that the appendix in the Bahdanau papers shows the details of how $\vec{c_t}$  is used:  https://arxiv.org/pdf/1409.0473.pdf#page=12, apparently roughly by adding (after projection) it to (projected versions of) $\vec{s_{i-1}}$ and $\vec{y_{i-1}}$ .  
  
  ---
  
  ### Answer by username_1 on 2022-10-08. Score: 0
  
  If our similarity function is defined as
  
  $$e^{t,t'}= f(y^t,h^{t'})$$
  for a $(t, t')$ pair, this would give you an attention weight/score for each hidden state $h^{t'}$. In this way you end up with $[0, ..., t-1]$ *weighted hidden states*, which you combine using a sum. This weighted sum is the "context vector"
  $$ c\_t = \sum\_{i=0}^{T}a\_{t,i}h\_i$$
  
  To answer your question, for each position you end up with a context vector $c\_t$ which fully replaces the hidden state $h^{t}$ at the exact same position of your computation graph.
  
  $$ h\_t \rightarrow c\_t,\ as\ \ LSTM\_{simple}\ \rightarrow\ LSTM\_{attentive} $$
  
  Hope it helps!
  
  **References**:   
  - Neural Machine Translation by Jointly Learning to Align and Translate, 2014 \
  
  > **Comments:**
  > **username_0**: This seems unlikely to me. What is this $h_t$ at the end of your post? It isn't an encoder state, so you must mean a decoder state $s_t$. Yet, Bahdanau on page 3 writes that $s_i = f(s_{i−1}, y_{i−1}, c_i)$, which suggests to me that $s_{t-1} \neq c_{t-1}$, otherwise Bahdanau would have written $s_i = f(c_{i−1}, y_{i−1}, c_i)$.  
  > **username_1**: Let me take a step back. In simple LSTM, can you define the difference between $h_t$ and $y_t$, so that i understand your pov better.  
  > **username_0**: It doesn't matter how I define these things. That equation is a literal quote from the Bahdanau paper linked in my question. It also doesn't matter whether it's with or without attention, because the Seq2Seq decoder LSTM feeds its own output $y_t$ back to its input regardless. $h_t$ are the encoder's hidden states. $s_t$ are the decoder's hidden states.  
  
  ---
  
  