Thanks for the cool proposal @graw !
The project is really cool  Just regarding the models, we don’t really have any of those implemented in Transformers, so this might take some time…Would it be sensible to pretrain a RoBERTa model + finetune it afterwards maybe?
Given that the project lasts only a week, maybe implementing + trying out all those models is a bit time-consuming