Roberta's Positional Embedding Offset
https://github.com/huggingface/transformers/blob/d4c2cb402d6674211726fd5f4803d1090664e438/src/transformers/modeling_bart.py#L754

https://github.com/huggingface/transformers/blob/d4c2cb402d6674211726fd5f4803d1090664e438/src/transformers/modeling_bart.py#L763

So this offset is added because the function `create_position_ids_from_input_ids` shifts the position ids by padding_idx + 1. However, I wonder if other models should also include this? 

https://github.com/huggingface/transformers/blob/d4c2cb402d6674211726fd5f4803d1090664e438/src/transformers/modeling_roberta.py#L54
For instance, when I am using `Longformer`, it looks like the offset is not added to `Roberta`, so I need to add such a offset to config.max_position_embeddings


That's certainly possible. As you can see from my comment, and PR #5188 , I don't fully understand the motivation for the offset. It is very tricky.<|||||>I figured out why. See here https://github.com/pytorch/fairseq/issues/1177
So basically the purpose is to make positional embedding = 0 on padding positions (positions where token is padding token), using the `padding_idx` parameter in torch.nn.Embedding. 

I think we can simply use masked_fill() to make positional embedding = 0 on padding positions, so the code is easier to understand (no need for the offset).<|||||>Exactly! 
Would love to do that, but the migration of the existing bart state dicts is non trivial, since they already store the extra position embedding. Even if we tracked down all bart models with `config.static_position_embeddings=False` and resized their positional embeddings, we would break code that is not up to date w master (lots of code).

So I think we must settle for documenting what is going on better in `LearnedPositionalEmbedding` and accept the unfortunate reality that we are stuck with the offset forever (or until we have some futuristic model hub tooling to version state dicts).<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
