@nbroad great points Nicolas! I am not sure if we will have the bandwidth to do pretraining (or intermediate training) but CORD-19 sounds quite nice! For finetuning, we definitely will need a mixing approach as explained in this COLAB 2. I think to save time we could very much follow that approach and mix in Covid QA, CDC QA, SQUAD and Trivia. We could use Seqio 2 for this mixing. Would you like to be a part of this project?