Oh but I am already applying a mask in the forward pass:
x = self.transformer(x, tgt, tgt_mask=generate_square_subsequent_mask(x.shape[0]), src_mask=generate_square_subsequent_mask(x.shape[0]))

From my understanding that generate_subsequent_mask() function hides the next token.