Alternative solution: not using the run-mlm.py script, but a code inspired from: lordtt13/COVID-SciBERT · Hugging Face 1 /  word-embeddings/COVID-SciBERT.ipynb at master · lordtt13/word-embeddings · GitHub 1 Thanks @lordtt13 !!
with:
 tokenizer = transformers.AutoTokenizer.from_pretrained('allenai/scibert_scivocab_uncased')
model = transformers.AutoModelWithLMHead.from_pretrained('allenai/scibert_scivocab_uncased').to('cuda')
Still would be interesting to know if the integration of scibert as a pre-trained model is planned for run_mlm.py