I found out my problem. My mask was the opposite of what it should have been. Pytorch’s Transformer model requires you to mask padded indices in a way that they become true while non-padded tokens are assigned a false value in the corresponding mask.