THERE ARE MANY CONSISTENT EXPLANATIONS OF UNLABELED DATA: WHY YOU SHOULD AVERAGE
Presently the most successful approaches to semi-supervised learning are based on consistency regularization, whereby a model is trained to be robust to small perturbations of its inputs and parameters. To understand consistency regularization, we conceptually explore how loss geometry interacts with training procedures. The consistency loss dramatically improves generalization performance over supervisedonly training; however, we show that SGD struggles to converge on the consistency loss and continues to make large steps that lead to changes in predictions on the test data. Motivated by these observations, we propose to train consistencybased methods with Stochastic Weight Averaging (SWA), a recent approach which averages weights along the trajectory of SGD with a modified learning rate schedule. We also propose fast-SWA, which further accelerates convergence by averaging multiple points within each cycle of a cyclical learning rate schedule. With weight averaging, we achieve the best known semi-supervised results on CIFAR-10 and CIFAR-100, over many different quantities of labeled training data. For example, we achieve 5.0% error on CIFAR-10 with only 4000 labels, compared to the previous best result in the literature of 6.3%.
This paper proposes to apply Stochastic Weight Averaging to the semi-supervised learning context. It makes an interesting argument that the semi-supervised MT/Pi models are especially amenable to SWA since they are empirically observed to traverse a large flat region of the weight space during the later stages of training. To speed up training, the authors propose fast-SWA.
Secition 3.2 is a little confusing. 
- If a random direction is, with high probability, not penalized, then why is it so flat along a random direction? Or is this simply an argument for why it is not guaranteed to be penalized, and therefore adversarial rays exist? I think the claim needs to be more precise (though it remains unclear how accurate the claim would be).
- I also think that there is maybe something special about measuring the SGD-SGD ray at epochs 170/180. It coincides with the regime of training where the signal is dominated by the consistency loss. Is it possible this somehow induces a near-linear path in the parameter space? I would be interested in seeing projections of other epoch’s SGD-SGD (e.g. 170/17x) vectors onto the 170/180 SGD-SGD ray and the extend to which they are co-linear. 
- It is also striking that traversing the SGD-SGD ray causes an error rate so similar to the adversarial ray for the supervised model; can the authors explain this phenomenon? 
- All this being said, I find the diversity argument compelling---though what would happen if we train the model even longer? Does it keep exploring?
- Overall, I am not sure how comfortable we should be with interpreting the SGD-SGD ray results. It is important that the authors provide a convincing argument for the interpretability of the SGD-SGD ray results, as this appears to be the key to the “large flat region” claim.
I think Mandt’s paper should be cited in-text, since this is what motivates Figure 2d.
Is the benefit of Fast-SWA’s fast convergence (to a competitive/better solution than SWA) unique to semi-supervised learning? Or can it be demonstrated by fully-supervised learning too? Given the focus on the semi-supervised regime, I would prefer if what the authors are proposing is, in some sense, special to the semi-supervised regime.
Table 1 is confusing to read. I just want to see a comparison between with and without using fast-SWA, *with all else kept equal*. Is the intention to compare “Previous Best CNN” and “Ours CNN”? Is this a fair comparison?
Pros:
+ Interesting story
+ Good empirical performance
Cons:
- Unclear whether the story is entirely correct
If the authors can provide a convincing case for the interpretability of the SGD-SGD results, I am happy to raise my score.