Answers.Body: <p>In seq2seq they model the joint distribution of whatever char/word sequence by decomposing it into time-forward conditionals:</p>
  <p><span class="math-container">\begin{align*}
  p(w_1,w_2, \dots,w_n) &amp;= \ p(w_1)*p(w_2|w_1) * \ ... \ * p(w_n|w_1, \dots,w_{n-1}) \\
   &amp;= \ p(w_1)*\prod_{i=2}^{n}p(w_i|w_{&lt;i})
  \end{align*}</span></p>
  <p>This can be sampled by sampling each of the conditional in ascending order. So, that's exactly what they're trying to imitate. You want the second output dependant on the <strong>sampled</strong> first output, not its distribution.</p>
  <p>This is why the hidden state is <strong>NOT</strong> good for modeling this setup because it is a latent representation of the distribution, not a sample of the distribution.</p>
  <p>Note: In training, they use ground-truth as input by default because it working under the assumption the model should've predicted the correct word, and, if it didn't, the gradient of the word/char level loss will reflect that (this is called teacher forcing and has a multitude of pitfalls).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-03T13:14:45.737Z
Answers.Text: can it be improved by entering position flags  for input like ( w,1) (x,2) ( y,3)???
Answers.UserDisplayName: user8426627
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-03T13:39:56.217Z
Answers.Text: @user8426627 maybe. probably dpends on the data and problem. Im assuming what you mean, is if you add some form of positional embedding will it improve the model? Positional embeddings are used alot recently because popular attention-only architectures dont embed position inherently, while RNN's do. So in this case, youd hope its not necessary because of the LSTM, but you never know, adding that representation may assist in the training. Could be a fun experiment :)
Answers.UserDisplayName: mshlis
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-03T11:51:06.437Z
Answers.LastActivityDate: 2021-12-12T12:08:59.740Z
Answers.LastEditDate: 2021-12-12T12:08:59.740Z
Answers.UserDisplayName: mshlis
Answers.Body: <blockquote>
    <p>Shouldn't the hidden state of the RNN from the previous timestamps be enough?</p>
  </blockquote>
  
  <p>It is theoretically enough to generate a sequence. However, allowing an input offers a couple of convenient extras:</p>
  
  <ul>
  <li><p>Training data for output sequences is used twice - once as input (as previous sequence data), once as target (to establish loss metric). This may help training process as the decoder trains both as a decoder to the new sequence type and as a predictive model over the output sequence semi-independently - i.e. weights from input to RNN layer are affected by error gradients separately to weights between previous hidden state and next state, although the two sets of weights together influence output and next state, so are not fully independent over a sequence.</p></li>
  <li><p>By allowing input of sequence so far generated, the decoder can work as a generator, where the next item in the sequence does not need to be the maximum probability item, but can be sampled or have rules applied. This allows for approaches such as <a href="https://medium.com/@dhartidhami/beam-search-in-seq2seq-model-7606d55b21a5" rel="nofollow noreferrer">BEAM search</a>, commonly used in machine translation, which maintains several potential outputs, selecting best one at the end.</p></li>
  </ul>
  
  <p>I have not done the experiment, but I suspect the first item results in faster and better generalisation. The second one is very convenient for natural language generation and similar problems.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-03T19:37:20.693Z
Answers.Text: @DuttA: Yes, with that intuition, it can also be considered a form of regularisation, because the weights have to satisfy two constraints instead of one.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-03T09:05:35.023Z
Answers.LastActivityDate: 2019-07-03T16:07:17.423Z
Answers.LastEditDate: 2019-07-03T16:07:17.423Z
Answers.UserDisplayName: Neil Slater
Answers.Body: <p>In the original seq2seq paper, they used two RNN, one for encoding and one for decoding. In the encoder they need to unroll the inputs to capture the time dependency. Now if we want to pass the hidden state from the encoder to the decoder that means that the decoder hidden state shape needs to match the encoder (aka same architecture). Since the architecture is the same, we can not directly generate a sequence of n samples within the decoder without unrolling it and you can not unroll it without an input.  </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-03T08:05:57.273Z
Answers.LastActivityDate: 2019-07-03T08:05:57.273Z
Answers.UserDisplayName: razvanc92
Body: <p>The basic seq-2-seq model consists of 2 parts: a recurrent encoder that compresses a sequence to a vector and decoder that unrolls the vector into the output sequence:</p>
  
  <p><a href="https://i.sstatic.net/IkEwP.png" rel="noreferrer"><img src="https://i.sstatic.net/IkEwP.png" alt="enter image description here"></a></p>
  
  <p>Why is the output, <code>w</code>, <code>x</code>, <code>y</code>, <code>z</code> of the decoder used as its input? Shouldn't the hidden state of the RNN from the previous timestamps be enough?</p>
  
Tags: <recurrent-neural-networks><sequence-modeling>
Title: In sequence-to-sequence, why is the output of the decoder used as its input?
UserDisplayName: user8426627
ThreadText: # In sequence-to-sequence, why is the output of the decoder used as its input?
  
  (Asked by: username_0 on 2019-07-02)
  
  The basic seq-2-seq model consists of 2 parts: a recurrent encoder that compresses a sequence to a vector and decoder that unrolls the vector into the output sequence:
  
  [![enter image description here](https://i.sstatic.net/IkEwP.png)](https://i.sstatic.net/IkEwP.png)
  
  Why is the output, `w`, `x`, `y`, `z` of the decoder used as its input? Shouldn't the hidden state of the RNN from the previous timestamps be enough?
  
  **Tags:** recurrent-neural-networks, sequence-modeling
  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_1 on 2019-07-03. Score: 3
  
  In seq2seq they model the joint distribution of whatever char/word sequence by decomposing it into time-forward conditionals:
  
  \begin{align\*}
  p(w\_1,w\_2, \dots,w\_n) &= \ p(w\_1)\*p(w\_2|w\_1) \* \ ... \ \* p(w\_n|w\_1, \dots,w\_{n-1}) \\
  &= \ p(w\_1)\*\prod\_{i=2}^{n}p(w\_i|w\_{<i})
  \end{align\*}
  
  This can be sampled by sampling each of the conditional in ascending order. So, that's exactly what they're trying to imitate. You want the second output dependant on the **sampled** first output, not its distribution.
  
  This is why the hidden state is **NOT** good for modeling this setup because it is a latent representation of the distribution, not a sample of the distribution.
  
  Note: In training, they use ground-truth as input by default because it working under the assumption the model should've predicted the correct word, and, if it didn't, the gradient of the word/char level loss will reflect that (this is called teacher forcing and has a multitude of pitfalls).
  
  > **Comments:**
  > **username_0**: can it be improved by entering position flags  for input like ( w,1) (x,2) ( y,3)???  
  > **username_1**: @username_0 maybe. probably dpends on the data and problem. Im assuming what you mean, is if you add some form of positional embedding will it improve the model? Positional embeddings are used alot recently because popular attention-only architectures dont embed position inherently, while RNN's do. So in this case, youd hope its not necessary because of the LSTM, but you never know, adding that representation may assist in the training. Could be a fun experiment :)  
  
  ---
  
  ### Answer by username_2 on 2019-07-03. Score: 1
  
  > Shouldn't the hidden state of the RNN from the previous timestamps be enough?
  
  It is theoretically enough to generate a sequence. However, allowing an input offers a couple of convenient extras:
  
  * Training data for output sequences is used twice - once as input (as previous sequence data), once as target (to establish loss metric). This may help training process as the decoder trains both as a decoder to the new sequence type and as a predictive model over the output sequence semi-independently - i.e. weights from input to RNN layer are affected by error gradients separately to weights between previous hidden state and next state, although the two sets of weights together influence output and next state, so are not fully independent over a sequence.
  * By allowing input of sequence so far generated, the decoder can work as a generator, where the next item in the sequence does not need to be the maximum probability item, but can be sampled or have rules applied. This allows for approaches such as [BEAM search](https://medium.com/@dhartidhami/beam-search-in-seq2seq-model-7606d55b21a5), commonly used in machine translation, which maintains several potential outputs, selecting best one at the end.
  
  I have not done the experiment, but I suspect the first item results in faster and better generalisation. The second one is very convenient for natural language generation and similar problems.
  
  > **Comments:**
  > **username_2**: @DuttA: Yes, with that intuition, it can also be considered a form of regularisation, because the weights have to satisfy two constraints instead of one.  
  
  ---
  
  ### Answer by username_3 on 2019-07-03. Score: 0
  
  In the original seq2seq paper, they used two RNN, one for encoding and one for decoding. In the encoder they need to unroll the inputs to capture the time dependency. Now if we want to pass the hidden state from the encoder to the decoder that means that the decoder hidden state shape needs to match the encoder (aka same architecture). Since the architecture is the same, we can not directly generate a sequence of n samples within the decoder without unrolling it and you can not unroll it without an input.
  
  
  ---
  
  