alvations:

If we don’t have a MAX_LENGTH pre-determined. What values should we initialize the attn layer with?


Do not initialize it to be fixed length. Don’t have a fixed matrix for self.att. In fact, self.att should NOT exist at all really. Remove that line of code and use the encoder source as the input. I’d recommend Luong’s attention so something like:
alpha_{s,t} = align( d_t, e_s ) ~ exp( < d_t, e_s > )

so have the encoders and decoders determine the weights. The whole point of attention is that the actual semantics of the encoding vector and target vector to determine the output of the RNN. Plus, attention makes things variable length using that. It’s ridiculous how the tutorial destroys all the advantages of RNNs in one go.