Interesting. But I don’t see a justification to use stale past states for forecasting, like in NLP tasks. Only capturing periodicities comes to mind, but this model is overkill for that problem.



 Gautham_Kumaran:

Currently, I see a slight improvement in my results when compared to a vanilla RNN approach.


Yes, but you don’t truly “compress” past data, so your model may learn simpler transition rules (e.g. AR(n)-like model).