Good catch. I think that in the decoder, one attends to all tokens (including padding), but as one needs to set the labels of the padding tokens to -100, they are not taken into account by the loss function.
But it’s weird indeed, as in the encoder one uses the attention mask to ignore padding tokens when calculating the attention scores.
cc @patrickvonplaten @valhalla