Perform pretraining on GLUE tasks with non-pretrained large model ?
I don’t know but multi-task learning may help ?
Additionally, since dataset size is smaller than those large text corpus for pretraining, stronger regularization (dropout, weight decay, gradient clip…) may help.