I’m curious if training a language model first on CORD-19 2 before doing fine-tuning would help.
Also since the COVID QA datasets are relatively small, would it be worthwhile to train on a generic QA dataset (e.g. SQuAD) before training on COVID QA? Or is there a way to mix the datasets to make a more robust model (i.e. for every 5 samples in COVID QA, throw in 1 sample of SQuAD)?
I know relatively nothing about fine-tuning QA models – maybe this approach is already well established as being intractable.