Padding Token in Transformer XL
I have sentences of varying lengths and I was wondering how to handle that as I could not see any padding token present. The index 0 refers to <eos> in the vocab, so any help on addition of padding would be appreciated
For these causal models that consider the left-context only, it's ok not to worry too much about padding since the attention modules only look to the previous tokens. Just be careful when you compute the loss to ignore the out-of-sentence-tokens (using loss functions `ignore_index` for instance).<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>Hi, I have a related doubt. In the example code [run_language_modeling.py](https://github.com/huggingface/transformers/blob/master/examples/run_language_modeling.py), GPT-2 and transformer-xl throw error due to lacking padding token.

Please check recent comments in this [Issue](https://github.com/huggingface/transformers/issues/3021)