Answers.Body: <p>Consider the task of sequence prediction, so you want to predict the next element of a sequence <span class="math-container">$e_t$</span> given the previous elements of this sequence <span class="math-container">$e_{t-1}, e_{t-2}, \dots, e_{1} = e_{t-1:1}$</span>. Teacher forcing is about forcing the predictions to be based on correct histories (i.e. the correct sequence of past elements) rather than predicted history (which may not be correct). To be more concrete, let <span class="math-container">$\hat{e}_{i}$</span> denote the <span class="math-container">$i$</span>th predicted element of the sequence and let <span class="math-container">$e_{i}$</span> be the corresponding ground-truth. Then, if you use teacher forcing, to predict <span class="math-container">$e_{t}$</span>, rather than using <span class="math-container">$\hat{e}_{t-1:1}$</span>, you would use <span class="math-container">$e_{t-1:1}$</span>.</p>
  
  <p>Recall that supervised learning can also be thought of as learning with a teacher. Hence the expression "teacher forcing", i.e. you <strong>force</strong> the predictions to be based on correct histories (the <strong>teacher</strong>'s labels). </p>
  
  <p>Of course, intuitively, teacher forcing should help to stabilize training, given that the predictions are not based on noisy or wrong histories.</p>
  
  <p>See also the blog post <a href="https://machinelearningmastery.com/teacher-forcing-for-recurrent-neural-networks/" rel="nofollow noreferrer">What is Teacher Forcing for Recurrent Neural Networks?</a> by Jason Brownlee.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-13T13:21:42.290Z
Answers.LastActivityDate: 2020-02-13T13:28:26.620Z
Answers.LastEditDate: 2020-02-13T13:28:26.620Z
Answers.UserDisplayName: nbro
Body: <p>In the paper <a href="https://arxiv.org/abs/1511.06279" rel="nofollow noreferrer">Neural Programmer-Interpreters</a>, the authors use the teacher forcing technique, but what exactly is it?</p>
  
Tags: <neural-networks><machine-learning><recurrent-neural-networks><definitions><teacher-forcing>
Title: What is teacher forcing?
UserDisplayName: nbro
ThreadText: # What is teacher forcing?
  
  (Asked by: username_0 on 2020-02-13)
  
  In the paper [Neural Programmer-Interpreters](https://arxiv.org/abs/1511.06279), the authors use the teacher forcing technique, but what exactly is it?
  
  **Tags:** neural-networks, machine-learning, recurrent-neural-networks, definitions, teacher-forcing
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2020-02-13. Score: 2 | Accepted answer
  
  Consider the task of sequence prediction, so you want to predict the next element of a sequence $e\_t$ given the previous elements of this sequence $e\_{t-1}, e\_{t-2}, \dots, e\_{1} = e\_{t-1:1}$. Teacher forcing is about forcing the predictions to be based on correct histories (i.e. the correct sequence of past elements) rather than predicted history (which may not be correct). To be more concrete, let $\hat{e}\_{i}$ denote the $i$th predicted element of the sequence and let $e\_{i}$ be the corresponding ground-truth. Then, if you use teacher forcing, to predict $e\_{t}$, rather than using $\hat{e}\_{t-1:1}$, you would use $e\_{t-1:1}$.
  
  Recall that supervised learning can also be thought of as learning with a teacher. Hence the expression "teacher forcing", i.e. you **force** the predictions to be based on correct histories (the **teacher**'s labels).
  
  Of course, intuitively, teacher forcing should help to stabilize training, given that the predictions are not based on noisy or wrong histories.
  
  See also the blog post [What is Teacher Forcing for Recurrent Neural Networks?](https://machinelearningmastery.com/teacher-forcing-for-recurrent-neural-networks/) by Jason Brownlee.
  
  
  ---
  
  