Probably a bug in XLMRobertaTokenizer
(Everything goes perfect when I did experiment with MultilingualBert, but seems only the base-model is released.)

When using XLM-R, the according tokenizer (XLMRobertaTokenizer) converts \<unk\> and every OOV token into id = 1. However, 1 should be the number of \<pad\>. (And the tokenizer can convert 1 to \<pad\>, 3 to \<unk\>).
Hi, if I am not wrong the pad should be 2 ?
at least the parameters tokenizer.pad_token_id for XLM-R 

EDIT: 2 is eos sorry.<|||||>Indeed, this is a bug that will be fixed when #3198 is merged. Thanks for letting us know.