Hi Alex, These were reasons for exploring this approach


The model is used to forecast multiple time-series (around 10K time-series), sort of like predicting the sales of each product in each store. I don’t want the overhead of training multiple models, so deep learning looked like a good choice. This also gives me the freedom to add categorical data as embeddings.


Initially, I tried to compress past data by only using the last hidden state and feed it into a Linear layer to forecast N timesteps, but this gave me poor results. Next, I tried to feed the results from all the GRU cells into a Linear layer to forecast N timesteps, this improved my results.


I saw two drawbacks in the previous architecture, The forecast of N timesteps is now independent of each other, which is not ideal, this made me choose an encoder-decoder architecture. Now, each decoder cell does not need the output from each cell in the encoder, and to address this some sort attention mechanism was needed. And I agree that the attention mechanism ended up capturing the periodicity.


This is the plot of the attention weights the model learned. The model is paying attention to timesteps from the distant past too, this is inline with what I thought would happen.
attention700×450 16.6 KB
One simplification I want to explore is to remove the attention layer, and just feed lagged timesteps to the decoder directly.
Let me know if I’m headed in the right direction.