  been swapped or not.
The library provides a version of the model for masked language modeling, token classification, sentence
classification, multiple choice classification and question answering.
RoBERTa
RoBERTa: A Robustly Optimized BERT Pretraining Approach, Yinhan Liu et al.
Same as BERT with better pretraining tricks:
dynamic masking: tokens are masked differently at each epoch, whereas BERT does it once and for all
no NSP (next sentence prediction) loss and instead of putting just two sentences together, put a chunk of
  contiguous texts together to reach 512 tokens (so the sentences are in an order than may span several documents)
train with larger batches
use BPE with bytes as a subunit and not characters (because of unicode characters)
The library provides a version of the model for masked language modeling, token classification, sentence
