Why does padding affect the embedding results for XLNet? Pre-padding returns different embeddings than post-padding. Which one should be used?
## ❓ Questions & Help

Hello, I am confused with different results of XLNet depending on padding. For Bert, padding doesn't affect the outputs, but for XLNet with **pre** padding (which I saw in https://github.com/huggingface/pytorch-transformers/blob/master/examples/run_glue.py#L281), returns very different results for the same sentence, with and without padding. The difference between no padding and post padding for XLNet returns similar results. Why did "pre" padding is used for run_glue.py? Does XLNet expect post padding or pre padding? Is there any document I am missing to clarify those important distinctions?

Here is a demonstration of the differences for pre, post padding for BERT and XLNet: https://colab.research.google.com/drive/1PCiU3icdfUB-nrLbrKAhgpePoIcFRlqX

Thanks,
Osman
I might be wrong, but intuitively I would say that that makes things easier. XLNet expects single sequences that look like this `tok1 tok2 ... SEP CLS`. So in contrast with BERT, the classification token is at the end of a sentence rather than beginning. This is before padding. So if you use post-padding, the position of the CLS element can differ for each element in your batch, but if you use pre-padding, then you can access the CLS element by its `-1` index.

That's not to say that it's not possible to retrieve the CLS element in XLNet when you've used post-padding. Something like this should work. Find the position (indices) where the input IDs are the classification token, then use those indices to slice the output.

```python
output = output[torch.where(input_ids == tokenizer.cls_token_id)]
```

If you've used pre-padding, this can be simplified to

```python
output = output[:, -1]
```<|||||>Hey @BramVanroy, thanks for the answer. You may be right that pre padding for XLNet makes things easier (i.e., getting `[cls]` token from the last index) but the question I want to be answered is not why we would like "pre" padding but why pre padding and post padding gives different answers. Maybe I should change the title further. If you see the notebook I shared, depending on padding you're getting different results.<|||||>Sorry, I fear I can't help with that. I am also wondering when padding is necessary and when it isn't. <|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>Maybe too late for this thread, but any answers to this issue?<|||||>Any answer on this issue?<|||||>Any answer on this issue?