@valhalla If I understand this correctly:


The input text will look like context: 42  is the answer to life, the universe and everything.  question: What is the answer to life, universe and everything ? answer:  42 


Mask out the question part so the new text will look like
context: 42  is the answer to life, the universe and everything.  <BIG MASK> answer: 42 


That is what gets fed as input text into the GPT2 model


Does this mean I define the labels into the model as the text that is masked?