Nothing is jumping out as wrong to me – except maybe I don’t see where are you initializing your hidden state. Are you getting an error or is your model just not converging?
Edit: I just realized I don’t see where you are masking your zero padded input – you don’t want to attend over zero pad values. Set any of your zero pad values to some really large negative number like -1e6 right before passing through softmax.