Padding in GPT-2  
How do I add padding in GTP2?
 I get something like this when I add zero in front to pad the sequences, but then I found out that 0 is actually not "[PAD]" but "!".
 [0, 0, 0, 0, 0, 0, 0, 0, 0, 1639, 481]

The zeros change the result quite a lot, not that it totally ruins it, but it makes the results less precise and usually order of the most frequent predicted words is altered.
So how to we add padding there.


I have tried to follow docs,  but I didn't find a way to add  something analog to attention mask BERT.


 




No padding implemented in GPT-2, you have to add implement your-self if you want e.g. by adding a special token but note that:
- GPT-2 doesn't like left side padding (doesn't mix well with a causal transformer having absolute positions)
- right-side padding is often not necessary (the causal mask means that right context is ignored anyway).<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>> * GPT-2 doesn't like left side padding (doesn't mix well with a causal transformer having absolute positions)

To those stumbling on this issue, this doesn't seem to be a problem anymore. [#3021](https://github.com/huggingface/transformers/issues/3021#issuecomment-1232149031)

