GPT2LMHeadModel with variable length batch input
I'm trying to repurpose the GPT2LMHeadModel for a seq2seq-like task, where I have an input prompt sequence of length L and I'm trying to ask the model to output a sequence to match a target sequence/sentence.
For a single input-output pair, I simply change the original code of 
`shift_logits = lm_logits[..., :-1, :].contiguous()`
to
`shift_logits = lm_logits[..., L-1:-1, :].contiguous()`

But I'm a bit lost on how can I do this for a batch of variable length input. Even if I pad the shorter sequences, I would need to shift the logits by different amount for each input. I'm also uncertain if I need to do something about the attention mask. Any tip is appreicated!
Have you tried concatenating the sequences into one long string and using a separator token without changing any of the code? You can then use a moving window of 1024 to train the model. You can make each step of the window start after an <|endoftext|> to ensure the primary sequence is not truncated.

You can then train on batches of these moving windows of 1024 (either moving a random # of tokens or to the next <|endoftext|> token)

E.g., An example input for 1024 may then look something like this: "Some sequence == Some other sequence <|endoftext|> Some sequence_2 == Some other sequence_2 <|endoftext|> Some sequence_3 == Some other sequence_3 <|endoftext|> Some sequence_4 == Some othe" (clipped on purpose to illustrate a point.)

Then you prompt with "Some sequence ==" and terminate generation/clip text on <|endoftext|>

The model is *very good* at learning like this. It is okay to have the window clip things off at the end.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
