Body: <p>I've been reading different papers which implements the <a href="https://arxiv.org/abs/1706.03762" rel="nofollow noreferrer">Transformer</a> for time series forecasting. Most of the them are claiming that the training time is significantly faster then using a normal RNN. From my understanding when training such a model, you can encode the input in parallel, but the decoding is still sequential unless you're using teacher-forcing. </p>
  
  <p>What makes the transformer faster than RNN in such a setting? Is there something that I am missing?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-10-25T01:19:23.347Z
Comments.Text: For those of you interested in the concept of teacher-forcing, have a look at the paper JT-VAE. Essentially, teacher-forcing is the trick of replacing the predictions with the actual ground-truth values, so that future predictions are based on correct past histories.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-10-25T06:46:45.943Z
Comments.Text: @nbro Thank you for your comment. I am aware how teacher-forcing works but if you're training a model in such a way the model is not gonna learn how to recover from a mistake. On the other hand if scheduled sampling is used i think the model might perform better, but this will significantly increase the training time.
Comments.UserDisplayName: razvanc92
Tags: <recurrent-neural-networks><transformer><prediction><attention><sequence-modeling>
Title: Why is the transformer for time series forecasting faster than RNN?
UserDisplayName: razvanc92
ThreadText: # Why is the transformer for time series forecasting faster than RNN?
  
  (Asked by: username_0 on 2019-10-24)
  
  I've been reading different papers which implements the [Transformer](https://arxiv.org/abs/1706.03762) for time series forecasting. Most of the them are claiming that the training time is significantly faster then using a normal RNN. From my understanding when training such a model, you can encode the input in parallel, but the decoding is still sequential unless you're using teacher-forcing.
  
  What makes the transformer faster than RNN in such a setting? Is there something that I am missing?
  
  **Tags:** recurrent-neural-networks, transformer, prediction, attention, sequence-modeling
  > **Comments:**
  > **username_1**: For those of you interested in the concept of teacher-forcing, have a look at the paper JT-VAE. Essentially, teacher-forcing is the trick of replacing the predictions with the actual ground-truth values, so that future predictions are based on correct past histories.  
  > **username_0**: @nbro Thank you for your comment. I am aware how teacher-forcing works but if you're training a model in such a way the model is not gonna learn how to recover from a mistake. On the other hand if scheduled sampling is used i think the model might perform better, but this will significantly increase the training time.  
  
  ---
  
  ## 0 answers
  
  