I tried this model too and the ‘bug’ i found is that when you create minibatches you sort the inputs with respect to their lengths, changing the order they are in.
My batch function is a bit different because it takes as param a list of indexes not inputs.
I fixed this like this(vec_q are my inputs):
vec_q, lengths, target, rindx = zip(*sorted(zip(vec_q, lengths, target, indexes), key=operator.itemgetter(1), reverse=True))
And then also returned rindx so i can use it later to put the predictions of the model where they should be.