Hi,
The reason autoregressive models like GPT2 are trained using a causal attention mask is because otherwise you “leak” information from the future. These models are trained to predict the next token, but if you let the model pay attention to the next token (while it needs to predict this), then that defeats the purpose of training the model.