Thanks for the response. I am using CTRL for my task and not much is available on it regarding how to finetune it properly. I have opened up a seperate thread but no response.
So attention_mask stays intact because we require the model to attend to the context. As you’re saying we do control the learning entirely by [-100] entries in labels, our attention_mask is basically a tensor filled with 1s. Coming to token_type_ids, I am making the parts that need to be generated 1s, rest 0s as per BERT documentation.
Now for labels, I’ve made everything [-100] except the parts that needs to be generated and shifting automatically happens inside the model. Having done this, I’m still not able to get good generations.
For CTRL, I don’t think we use a SEP token, I didn’t see it being used in original repo. Also the tokenizer (BPE based) is working quite differently i.e it is not adding special tokens as one would expect.
So my sentences were like [Custom control Code] [Sentence1] [Sentence2].
The documentation says this
The PyTorch models can take the past as input, which is the previously computed key/value attention pairs. but this is only applicable during inference right ?