<p>I've been reading different papers which implements the <a href="https://arxiv.org/abs/1706.03762" rel="nofollow noreferrer">Transformer</a> for time series forecasting. Most of the them are claiming that the training time is significantly faster then using a normal RNN. From my understanding when training such a model, you can encode the input in parallel, but the decoding is still sequential unless you're using teacher-forcing. </p>

<p>What makes the transformer faster than RNN in such a setting? Is there something that I am missing?</p>

CC BY-SA 4.0
2019-10-25T01:19:23.347Z
For those of you interested in the concept of teacher-forcing, have a look at the paper JT-VAE. Essentially, teacher-forcing is the trick of replacing the predictions with the actual ground-truth values, so that future predictions are based on correct past histories.
nbro
CC BY-SA 4.0
2019-10-25T06:46:45.943Z
@nbro Thank you for your comment. I am aware how teacher-forcing works but if you're training a model in such a way the model is not gonna learn how to recover from a mistake. On the other hand if scheduled sampling is used i think the model might perform better, but this will significantly increase the training time.
razvanc92
<recurrent-neural-networks><transformer><prediction><attention><sequence-modeling>
Why is the transformer for time series forecasting faster than RNN?
razvanc92
# Why is the transformer for time series forecasting faster than RNN?

(Asked by: username_0 on 2019-10-24)

I've been reading different papers which implements the [Transformer](https://arxiv.org/abs/1706.03762) for time series forecasting. Most of the them are claiming that the training time is significantly faster then using a normal RNN. From my understanding when training such a model, you can encode the input in parallel, but the decoding is still sequential unless you're using teacher-forcing.

What makes the transformer faster than RNN in such a setting? Is there something that I am missing?

**Tags:** recurrent-neural-networks, transformer, prediction, attention, sequence-modeling

> **Comments:**
> **username_1**: For those of you interested in the concept of teacher-forcing, have a look at the paper JT-VAE. Essentially, teacher-forcing is the trick of replacing the predictions with the actual ground-truth values, so that future predictions are based on correct past histories.  
> **username_0**: @username_1 Thank you for your comment. I am aware how teacher-forcing works but if you're training a model in such a way the model is not gonna learn how to recover from a mistake. On the other hand if scheduled sampling is used i think the model might perform better, but this will significantly increase the training time.  

---

## 0 answers

