Answers.Body: <p>That drawing it's a bit oversimplified. Check this <a href="https://github.com/bentrevett/pytorch-seq2seq/blob/master/1%20-%20Sequence%20to%20Sequence%20Learning%20with%20Neural%20Networks.ipynb" rel="nofollow noreferrer">blog</a> for a better explanation and implementation details. I'll refer to the image they have to answer:</p>
  <ul>
  <li>the yellow boxes represent embedding layers, required to convert words in numbers</li>
  <li>the green boxes represent the unfolded encoder</li>
  <li><strong>the red box represent the context vector</strong>, i.e. the vector you're looking for. Note that is just the final vector you obtain by applying the encoder to a sequence of words. For this reason some people prefer to draw directly a line to the decoder part, without drawing the final vector explicitly.</li>
  <li>the blue boxes represent the unfolded decoder.</li>
  <li>the purple boxes represent the linear layer used to predict a final word for the decoder hidden state.</li>
  </ul>
  <p><a href="https://i.sstatic.net/veic6.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/veic6.png" alt="enter image description here" /></a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-11T10:29:00.170Z
Answers.Text: The blog is reversing the input order and the paper also says that they reversed the order. But at the diagram you mentioned it's not in reverse order. Is it just an error or intentional do you have any ideas? For example it supposed to be morgen guten I guess.
Answers.UserDisplayName: J.Smith
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-11T10:44:50.460Z
Answers.Text: yes, it's suppose to be <sos> morgen guten <eos>. I guess it's probably an error, or they left the correct order in the diagram just for readability.
Answers.UserDisplayName: Edoardo Guerriero
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-11T08:26:09.973Z
Answers.LastActivityDate: 2021-11-11T08:26:09.973Z
Answers.UserDisplayName: Edoardo Guerriero
Body: <p>I'm reading <a href="https://arxiv.org/abs/1409.3215" rel="nofollow noreferrer">Sequence to Sequence Learning with Neural Networks</a> and there's a thing that I couldn't quite grasp.</p>
  <p>Paper says the encoder outputs a vector to be fed to the decoder. More precisely</p>
  <blockquote>
  <p>Our method uses a multilayered Long Short-Term Memory (LSTM) to map the input sequence to a vector of a fixed dimensionality, and then another deep LSTM to decode the target sequence from the vector</p>
  </blockquote>
  <p>However, when I look at the diagram:</p>
  <p><a href="https://i.sstatic.net/NGSf3.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/NGSf3.png" alt="enter image description here" /></a></p>
  <p>there's no such vector here. What I understand from this diagram is decoder RNN takes the weights of the last encoder cell as an input.</p>
  <p>Which one is correct? can you explain?</p>
  <p>Stanford notes put it as</p>
  <blockquote>
  <p>The final hidden state of the cell will then become C</p>
  </blockquote>
  <p>So, is there no vector?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-11T08:55:46.490Z
Comments.Text: `w` here does not represent weights, in case you were thinking it does.
Comments.UserDisplayName: David Hoelzer
Tags: <recurrent-neural-networks><papers><seq2seq>
Title: Does Seq2Seq decoder take a special vector or the weights of the last encoder cell as an output?
UserDisplayName: J.Smith
ThreadText: # Does Seq2Seq decoder take a special vector or the weights of the last encoder cell as an output?
  
  (Asked by: username_0 on 2021-11-10)
  
  I'm reading [Sequence to Sequence Learning with Neural Networks](https://arxiv.org/abs/1409.3215) and there's a thing that I couldn't quite grasp.
  
  Paper says the encoder outputs a vector to be fed to the decoder. More precisely
  
  > Our method uses a multilayered Long Short-Term Memory (LSTM) to map the input sequence to a vector of a fixed dimensionality, and then another deep LSTM to decode the target sequence from the vector
  
  However, when I look at the diagram:
  
  [![enter image description here](https://i.sstatic.net/NGSf3.png)](https://i.sstatic.net/NGSf3.png)
  
  there's no such vector here. What I understand from this diagram is decoder RNN takes the weights of the last encoder cell as an input.
  
  Which one is correct? can you explain?
  
  Stanford notes put it as
  
  > The final hidden state of the cell will then become C
  
  So, is there no vector?
  
  **Tags:** recurrent-neural-networks, papers, seq2seq
  
  > **Comments:**
  > **username_1**: `w` here does not represent weights, in case you were thinking it does.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-11-11. Score: 1 | Accepted answer
  
  That drawing it's a bit oversimplified. Check this [blog](https://github.com/bentrevett/pytorch-seq2seq/blob/master/1%20-%20Sequence%20to%20Sequence%20Learning%20with%20Neural%20Networks.ipynb) for a better explanation and implementation details. I'll refer to the image they have to answer:
  
  * the yellow boxes represent embedding layers, required to convert words in numbers
  * the green boxes represent the unfolded encoder
  * **the red box represent the context vector**, i.e. the vector you're looking for. Note that is just the final vector you obtain by applying the encoder to a sequence of words. For this reason some people prefer to draw directly a line to the decoder part, without drawing the final vector explicitly.
  * the blue boxes represent the unfolded decoder.
  * the purple boxes represent the linear layer used to predict a final word for the decoder hidden state.
  
  [![enter image description here](https://i.sstatic.net/veic6.png)](https://i.sstatic.net/veic6.png)
  
  > **Comments:**
  > **username_0**: The blog is reversing the input order and the paper also says that they reversed the order. But at the diagram you mentioned it's not in reverse order. Is it just an error or intentional do you have any ideas? For example it supposed to be morgen guten I guess.  
  > **username_2**: yes, it's suppose to be <sos> morgen guten <eos>. I guess it's probably an error, or they left the correct order in the diagram just for readability.  
  
  ---
  
  