I am using a dataset from dataset library for text generation task. For each example in dataset, I want to provide sentence1 and label as context, and allow network to generate sentence2 and this should be used for loss calculation. In the run_clm script, I’m not able to find this distinction as to what is being used as context.
Based on my understanding attention_mask and token_type_ids are responsible for controlling what network sees and what not. So from this, I want label+sentence1 to have 0s and sentence2 to have 1s in attention_mask.
def tokenize_function(examples):
    tokenized_input = tokenizer(LABELS[examples['label']], examples['sentence1'])
    # not sure how to manage examples['sentence2']
    return tokenized_input


I want the output to be [CLS] [SEP] label [SEP] sentence 1 [SEP] sentence 2 (to be generated) [SEP]
Can anyone clarify ?