Answers.Body: <p>The reason why you would use the ground truth as input to the decoder is to learn the testing distribution. From what I've seen so far, most of the papers are using scheduled sampling (Bengio et al.). Meaning that you will introduce a new term <span class="math-container">$p$</span>, which will be the probability of the network to feed as input its own prediction. Initially, <span class="math-container">$p$</span> will be very small, so that the network will use the ground truth, but later as more iterations pass by, the probability will increase and the network will start using its own predictions.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-13T06:23:45.173Z
Answers.LastActivityDate: 2023-10-24T16:33:44.030Z
Answers.LastEditDate: 2023-10-24T16:33:44.030Z
Answers.UserDisplayName: razvanc92
Body: <p>I've read about seq2seq for time-series and it seemed really promising, but, when I went to implement it, all the tutorials I've found use the correct output as input to the decoder phase during training, instead of using the actual prediction made by the cell before it. Is there a reason why not do the latter?</p>
  <p>I've been using the tutorial from <a href="https://weiminwang.blog/2017/09/29/multivariate-time-series-forecast-using-seq2seq-in-tensorflow/" rel="nofollow noreferrer">here</a></p>
  <p>But all the other tutorials that I've found followed the same principle.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-09-15T08:48:43.857Z
Comments.Text: This is called "teacher forcing". See here: https://en.m.wikipedia.org/wiki/Teacher_forcing
Comments.UserDisplayName: pi-tau
Tags: <deep-learning><time-series><sequence-modeling><seq2seq><teacher-forcing>
Title: Why feeding the correct output as input during training of seq2seq models?
UserDisplayName: Ramon Dettmer
ThreadText: # Why feeding the correct output as input during training of seq2seq models?
  
  (Asked by: username_0 on 2019-06-13)
  
  I've read about seq2seq for time-series and it seemed really promising, but, when I went to implement it, all the tutorials I've found use the correct output as input to the decoder phase during training, instead of using the actual prediction made by the cell before it. Is there a reason why not do the latter?
  
  I've been using the tutorial from [here](https://weiminwang.blog/2017/09/29/multivariate-time-series-forecast-using-seq2seq-in-tensorflow/)
  
  But all the other tutorials that I've found followed the same principle.
  
  **Tags:** deep-learning, time-series, sequence-modeling, seq2seq, teacher-forcing
  
  > **Comments:**
  > **username_1**: This is called "teacher forcing". See here: https://en.m.wikipedia.org/wiki/Teacher_forcing  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2019-06-13. Score: 1
  
  The reason why you would use the ground truth as input to the decoder is to learn the testing distribution. From what I've seen so far, most of the papers are using scheduled sampling (Bengio et al.). Meaning that you will introduce a new term $p$, which will be the probability of the network to feed as input its own prediction. Initially, $p$ will be very small, so that the network will use the ground truth, but later as more iterations pass by, the probability will increase and the network will start using its own predictions.
  
  
  ---
  
  