veronica320:

sentences are semantically likely vs. unlikely given a previous prompt.


So you are not necessarily interested in the likelihood of the sentence in itself, but given a previous prompt/sentence? In that case the NLI benchmark might be related (though not exactly what you want). You can find finetuned models on the hub.
But to stick to your approach, I would not make use of MLM models for this use case. I don’t see how they relate - they are not generation (decoder) models. More interesting (imo) would be to compare GPT-2 with GPT-3 (you can request beta API access) in this context, or GPT-J.