Hi all!
I’m hoping to use pretrained BERT/RoBERTa for language modeling, i.e. scoring the likelihood of sentences. There have been quite a few blog posts/issues on this, but no obvious consensus yet. I attempted an implementation with BertLMHeadModel and RobertaForCausalLM here 6, which had weird issues like:

RoBERTa has super large perplexity values, and
BERT cannot correctly compare the relative perplexity of simple sentences.
(Please see more details in the Github issue above.)

@gugarosa kindly suggests that I shouldn’t evaluate pretrained BERT/RoBERTa directly, but should train them with causal LM objective beforehand. However, given the size of their pretraining data, it’s unlikely to retrain them myself. Is there any existing checkpoints that I can use directly? Or, would you recommend any other models (e.g. BertForMaskedLM?) or evaluation metrics (other than perplexity) instead?
Thanks in advance!