fix padding_idx of RoBERTa model
The padding index of the pretrained RoBERTa model is 1, and 0 is assigned to `<s>` token. The padding index of the current RoBERTa model is set to be 0, therefore `<s>` is treated as padding.
This PR aims to fix this problem.
LGTM but let's have @julien-c or @LysandreJik confirm<|||||>lgtm too<|||||>It would be appreciated if you review this PR! @LysandreJik <|||||>@ikuyamada Out of curiosity, in which cases did you need to specify this `padding_idx`? It shouldn't have any impact on the inference so are you training a model from scratch? (with @LysandreJik)<|||||>Merged in https://github.com/huggingface/transformers/commit/a6a6d9e6382961dc92a1a08d1bab05a52dc815f9<|||||>big drawback is that we initialize the embeddings multiple times.

@ikuyamada do you have an idea to improve this?<|||||>Thank you for merging this PR!

> Out of curiosity, in which cases did you need to specify this padding_idx? It shouldn't have any impact on the inference so are you training a model from scratch?

In my understanding, the embedding corresponding to `padding_idx` is not updated while training (pre-training or fine-tuning). Because the embedding of the token `<s>` may play some roles for computing contextualized embeddings for other tokens, and the output embedding of the `<s>` token is used for computing a feature vector for some fine-tuning tasks, I think the embedding should be updated while training.

> big drawback is that we initialize the embeddings multiple times.
> @ikuyamada do you have an idea to improve this?

We can avoid this by removing [the constructor call of the `BertEmbeddings`](https://github.com/huggingface/transformers/blob/master/transformers/modeling_roberta.py#L44) and simply initialize the `token_type_embeddings`, `LayerNorm`, and `dropout` in the constructor of the `RobertaEmbeddings`.
If you prefer this implementation, I will create a PR again! :)
@julien-c <|||||>> In my understanding, the embedding corresponding to padding_idx is not updated while training (pre-training or fine-tuning). Because the embedding of the token `<s>` may play some roles for computing contextualized embeddings for other tokens, and the output embedding of the `<s>` token is used for computing a feature vector for some fine-tuning tasks, I think the embedding should be updated while training.

Yes you are correct.

> If you prefer this implementation, I will create a PR again! :)

That would be great, thank you.