Since GPT-2 is an auto-regressive decoder, we need to feed both the context and target as the input text.

In the the script I see result["labels"] = result["input_ids"].copy which just means we are taking everything as label and then perform CLM on both sentences

Since it’s a CLM, during training it predicts next token for every token in the input.
So we pass the labeld=input_ids and then lables are shifted to right inside the model 2.
i.e if input_ids = [BOS] tok1 tok2 tok3....
then labels= tok1 tok2 tok3
and all the tokens in the labels are considered when calculating the loss.
Now if we want to use a CLM for tasks that require context and target (let’s say a summarization task) and we only want the model to generate a target (summary) then
During Trainig:
we need to concatenate both the context (source_text) and target (summary) as  one single input text
i.e input = [BOS] source_text [SEP] summary
and prepare labels such that tokens corresponding to the context (source text) will have -100 as the value. This way we ensure that the loss is only calculated for summary tokens so the model will (hopefully) only learn to generate summary (or target).
And During inference
we would feed the source text with sep and ask the model to generate the next tokens which will be summary in this case i.e
input = [BOS] source text [SEP]
Also, the attention_mask here is the same as attention_mask in every other model, it specifies which tokens in the input should be attended and which should be ignored.
For reference you could refer to this conv ai example which uses this approach for Conversational agent with GPT2
  
      github.com
  
  
    huggingface/transfer-learning-conv-ai/blob/d4c760731e8f062abd3d61ccdc8fa29ccb5e3485/train.py#L56-L69 2

def build_input_from_segments(persona, history, reply, tokenizer, lm_labels=False, with_eos=True):    """ Build a sequence of input from 3 segments: persona, history and last reply. """    bos, eos, speaker1, speaker2 = tokenizer.convert_tokens_to_ids(SPECIAL_TOKENS[:-1])    sequence = [[bos] + list(chain(*persona))] + history + [reply + ([eos] if with_eos else [])]    sequence = [sequence[0]] + [[speaker2 if (len(sequence)-i) % 2 else speaker1] + s for i, s in enumerate(sequence[1:])]    instance = {}    instance["input_ids"] = list(chain(*sequence))    instance["token_type_ids"] = [speaker2 if i % 2 else speaker1 for i, s in enumerate(sequence) for _ in s]    instance["mc_token_ids"] = len(instance["input_ids"]) - 1    instance["lm_labels"] = [-100] * len(instance["input_ids"])    if lm_labels:        instance["lm_labels"] = ([-100] * sum(len(s) for s in sequence[:-1])) + [-100] + sequence[-1][1:]    return instance



  
  
    
    
  
  


Hope this helps