I know this is an old post, but in case you are still interested there is an ongoing pull request for it here: Add support for XLM-R XL and XXL models by modeling_xlm_roberta_xl.py by Soonhwan-Kwon · Pull Request #13727 · huggingface/transformers · GitHub 44