Gautham_Kumaran:

I don’t completely understand this. My understanding of attention mechanism until now is that it looks at the output of past timesteps to determine the importance of each encoder output in determining the current decoder output. Is there an alternative? If it’s not looking at explicit memory, what will the attention weights be applied to?


What I meant, is that your decoder receives some initial hidden state, that can by itself express pretty complex dynamics. But you’re using more complex way to produce attention_weights, interleaving their generation and hidden state transitions. My theory is that as attention_weight are just query keys, you can generate them in isolation, i.e. without reading “explicit memory” - advantage is that this is doable with faster standard rnn cells. Well, that may be too different from what you’re doing…