Hi @Gautham_Kumaran, just saw your post at TowardsDataScience. Looks great!
However, I have a concern regarding your proposed architecture. In the diagram you shared, Assuming that t-1 is the last known step at inference time:
You are feeding the last observed target t-1 twice: once into the encoder and another time into the decoder.
This means that, essentially, your hidden state (or context vector, the yellow box called hidden in your diagram) already has info regarding t-1. However, you are re-feeding t-1 yet again to the decoder - which makes sense, given that you need to give something as decoder input to output the prediction for t. However, wouldn’t it be better just to only feed the encoder up to t-2?
This way you avoid giving essentially the same info to your decoder twice in the first decoder step, hence at all times the hidden state in both the encoder and the decoder is coherent content-wise, and contains the internal representation for all seen (or inferred) timesteps only once.
This is the way it’s done in the DeepAR paper for instance. The only change is that in DeepAR the encoder and the decoder are actually the same module, sharing exact architecture and weights.
With that said, I have also seen publications where the input for the decoder is always a copy of the encoder’s output vector as a constant for all decoder timesteps… So perhaps as long as your loss and training procedure are OK, the network learns to accomodate to whatever input you feed it…
What do you think?