Load pretrained roberta model from fairseq?
# ❓ Questions & Help
Would it be possible to load pretrained model of roberta-base that is trained by myself using fairseq, following the instructions in https://github.com/pytorch/fairseq/blob/master/examples/roberta/README.pretraining.md?

Since the current transformers library is not suitable for pretraining from scratch, I think it would be nice to be able to load pre-trained model trained on fairseq. I think it might be possible but I am not sure how the current transformers' roberta pretrained model is translated/loaded?

thanks!
Try https://github.com/huggingface/transformers/blob/master/src/transformers/convert_roberta_original_pytorch_checkpoint_to_pytorch.py

You might have to make some modifications, I have never tried this. Good luck!
<|||||>huggingface/transformers can be used to train from scratch. See [how to train a LM from scratch](https://huggingface.co/blog/how-to-train).<|||||>Feel free to open another issue if more specific <|||||>related official blog: "Porting fairseq wmt19 translation system to transformers" by @stas00 
https://huggingface.co/blog/porting-fsmt

Might be able to convert fairseq language models following similar steps.<|||||>> Try https://github.com/huggingface/transformers/blob/master/src/transformers/convert_roberta_original_pytorch_checkpoint_to_pytorch.py
> 
> You might have to make some modifications, I have never tried this. Good luck!

Hey :) I'm interested in this but the link doesn't seem to work.<|||||>when pasting links to a repo one needs to hit `y` to get a fixed link to the current revision which would never break (as long as the repo is in place), here you go:

https://github.com/huggingface/transformers/blob/7c6d63298f27b4a386f4603262a4603a2a6bf057/src/transformers/models/roberta/convert_roberta_original_pytorch_checkpoint_to_pytorch.py
