I’ve decided that there’s no bug. What’s happening is the model trained on a bunch of longer sentences and thus learned to look at those places, even if the new input is shorter. The solution to this would be to train on shorter sentences first, and longer ones later, with some slight variability.