Very cool project! I think we should try to keep the scope as small as possible → so maybe:

PreTrain RoBERTa on permuted words
Evaluate “permuted words” pretrained RoBERTa against “real” pretrained RoBERTa on PAWS?

What do you think @andreaparker ?
Anyone who would like to join?