Why the RoBERTa's max_position_embeddings size is 512+2=514?
## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
When I see the code of Roberta, I have a question about the padding_idx = 1, I don't know very well. And the comment is still confused for me.
What's your precise question?<|||||>> What's your precise question?

the self.padding_idx's meaning in modeling_roberta.py<|||||>It's the position of the padding vector. It's not unique to RoBERTa but far more general, especially for embeddings. Take a look at [the PyTorch documentation](https://pytorch.org/docs/stable/nn.html#embedding).<|||||>> It's the position of the padding vector. It's not unique to RoBERTa but far more general, especially for embeddings. Take a look at [the PyTorch documentation](https://pytorch.org/docs/stable/nn.html#embedding).

I know that, but I confuse about why there is 1 and the \<s\> is 0, is it ignore and why the max_position_embeddings size is 512+2=514?<|||||>Because that's their index [in the vocab](https://s3.amazonaws.com/models.huggingface.co/bert/roberta-base-vocab.json). The max_position_embeddings size is indeed 514, I'm not sure why. The tokenizer seems to handle text correctly with a max of 512. Perhaps someone of the developers can help with that. I would advise you to change the title of your topic.

https://github.com/huggingface/transformers/blob/ae50ad91ea2fedb64ecd2e7c8e2d0d4778dc03aa/transformers/tokenization_roberta.py#L84-L85<|||||>@LysandreJik can chime in if I’m wrong, but afaik `max_position_embeddings` is just the name of the variable that we use to encode the size of the embedding matrix. Max_len is correctly set to 512.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>Answer here in case anyone from the future is curious: https://github.com/pytorch/fairseq/issues/1187<|||||>> Answer here in case anyone from the future is curious: [pytorch/fairseq#1187](https://github.com/pytorch/fairseq/issues/1187)

@morganmcg1 Tks for this, was getting all kinds of CUDA errors because i setted `max_position_embeddings=512`, now that i setted 514 it's running ok...