of positional embeddings, the model has language embeddings. When training using MLM/CLM, this gives the model an
indication of the language used, and when training using MLM+TLM, an indication of the language used for each part.
The library provides a version of the model for language modeling, token classification, sentence classification and
question answering.
XLM-RoBERTa
Unsupervised Cross-lingual Representation Learning at Scale, Alexis Conneau et
al.
Uses RoBERTa tricks on the XLM approach, but does not use the translation language modeling objective. It only uses
masked language modeling on sentences coming from one language. However, the model is trained on many more languages
(100) and doesn't use the language embeddings, so it's capable of detecting the input language by itself.
