In natural language processing, it has been observed recently that generalization could be greatly improved by finetuning a large-scale language model pretrained on a large unlabeled corpus. Despite its recent success and wide adoption, finetuning a large pretrained language model on a downstream task is prone to degenerate performance when there are only a small number of training instances available. In this paper, we introduce a new regularization technique, to which we refer as "mixout", motivated by dropout. Mixout stochastically mixes the parameters of two models. We show that our mixout technique regularizes learning to minimize the deviation from one of the two models and that the strength of regularization adapts along the optimization trajectory. We empirically evaluate the proposed mixout and its variants on finetuning a pretrained language model on downstream tasks. More specifically, we demonstrate that the stability of finetuning and the average accuracy greatly increase when we use the proposed approach to regularize finetuning of BERT on downstream tasks in GLUE.
The authors introduce a new regularization technique for the specific task of finetuning models. It's inspired by dropout and stochastically mixes source and target weights in order to avoid moving the parameters towards 0.
The authors provide a theoretical justification as to why mixout would do useful things in the convex case and then demonstrate empirically that using it achieves good accuracies on some downstream, finetuned, non-convex-loss-utilizing tasks. Their experiments incorporate both small models with good analysis (ie, sec 4) as well as larger, real-world models (sect 5). The paper is in general well-written.
I have a few concerns that I would like to see addressed:
1a. For starters, all the theoretical motivation describes a particular way in which mixout is supposed to aid in downstream tasks for the case of convex functions, but there are no experiments that match their assumptions and which demonstrate this is the actual behavior we see. It would be nice to see results which demonstrate the theory.
1b. In few of the prsented empirical experiments is it the case that the use of mixout by itself is useful.  Why does mixout have to be coupled with other regularization techniques? There is little analysis given here, either empirical or theoretical. 
2. Why are there only 4 GLUE tasks reported? Devlin 2018 reports on all but WNLI.
3. The choice of hyperparameters for GLUE in sect 5 is a bit misleading. Devlin 2018 chose those parameters to get the maximum scores on downstream tasks; the metric they use is max score. However, the authors want to instead discuss the average score against a set of random restarts, perhaps because the max scores using their method aren't terribly different from the baselines. 
Therefore, a more extensive hyperparameter sweep should have been run for the baselines: they should have been re-tuned for the average score if that is the metric the authors wish to use. Instead, the authors only used one task, RTE, to find baseline hyperparameters.