Thanks for the follow up. Is the higher loss a problem, since you mentioned that your attention model is in fact producing better predictions?
Did you compare the output logits to check my claim, that they might be closer to each other?