Thanks Alex. I can understand your point on explicit memory. I’ll read up on this part.



 googlebot:

I could imagine simple GRU producing attention weights without consulting “explicit memory”.
And single 4d attention query after that…


I don’t completely understand this. My understanding of attention mechanism until now is that it looks at the output of past timesteps to determine the importance of each encoder output in determining the current decoder output. Is there an alternative? If it’s not looking at explicit memory, what will the attention weights be applied to?
And I’m exploring stuff on arxiv on this topic too, there are advancements in the attention mechanism itself, like adding convolutions to it - https://arxiv.org/pdf/1809.04206v3.pdf 19 and thanks for validating the coding part too