Gautham_Kumaran:

Initially, I tried to compress past data by only using the last hidden state and feed it into a Linear layer to forecast N timesteps, but this gave me poor results. Next, I tried to feed the results from all the GRU cells into a Linear layer to forecast N timesteps, this improved my results.


You can transform last hidden state and make it initial state for another forecasting RNN. You may feed it some future time measure, last output (if you do one step at a time) or even nothing at all. That’s decoder too.



 Gautham_Kumaran:

One simplification I want to explore is to remove the attention layer, and just feed lagged timesteps to the decoder directly.


Yes, something like that.
About attention… Contrived example - if encoder captures speed & acceleration, there is no need to look back. If you want to replicate stationary series, maybe with some variations - yea, it may be easier with attention. But then it eats a lot of extra resources too…