Thanks @spro!
Just one question, is there a good way to mask the attention? Even using the attention from the paper from Manning you mentioned, if the sentences in a batch are of different lengths, you will need to mask some positions. Right now I am directly filling the attention matrix (before applying softmax) with -float(“inf”), but this seems a bit hacky.