Hi @BramVanroy, thanks for your reply! Regarding your questions:

My end goal is to score the likelihood of sentences using BERT/RoBERTa on my custom dataset. My dataset can be thought of as a semantic version of CoLA, where models decide which sentences are semantically likely vs. unlikely given a previous prompt.
I’m not really insisting on the CLM objective; instead, I’m just interested in seeing how all kinds of language models (including BERT, RoBERTa, GPT…) can perform on my dataset. I have already included the results with GPT2 and there’s no problem with it. I hope to also get results from MLM-based models as well.
CoLA has a training set, but my dataset unfortunately doesn’t – that’s why I’m hoping to evaluate an out-of-the-shelf model.