I am not sure what you mean by “switch the attention set”. It applies the attention mask to hide future tokens if it’s what you mean (otherwise you would see a perplexity of 0 or 1 at the end of training).