Ah yes, sorry for my misunderstanding. So we mask out the parts we want to predict by setting the attention_mask of those tokens to 0.
With these tokens masked in attention_mask, do we then pass it and the input string to GPT2 and train it with the language model head with no labels?