Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs
The loss functions of deep neural networks are complex and their geometric properties are not well understood. We show that the optima of these complex loss functions are in fact connected by simple curves over which training and test accuracy are nearly constant. We introduce a training procedure to discover these high-accuracy pathways between modes. Inspired by this new geometric insight, we also propose a new ensembling method entitled Fast Geometric Ensembling (FGE). Using FGE we can train high-performing ensembles in the time required to train a single model. We achieve improved performance compared to the recent state-of-the-art Snapshot Ensembles, on CIFAR-10, CIFAR-100, and ImageNet.
Update after author response:  Thank you for the response. Additional details that the curves between the local optima are not unique would be also interesting to see. Nice work!  Summary: This paper first shows a very interesting finding on the loss surfaces of deep neural nets, and then presents a new ensembling method called Fast Geometric Ensembling (FGE). Given two already well trained deep neural nets (with no limitations on their architectures, apparently), we have two sets of weight vectors w1 and w2 (in a very high-dimensional space). This paper states a (surprising) fact that for given two weights w1 and w2, we can (always?) find a connecting path between w1 and w2 where the training and test accuracy is nearly constant. Figure 1 demonstrates this, and Left is the training accuracy plot on the 2D subspace passing independent weights w1, w2, w3 of ResNet-164 (from different random starts); whereas Middle and Right are the 2D subspace passing independent weights w1, w2 and one bend point w3 on the curve (Middle: Bezier, Right: Polygonal chain). Learning the curves with one bend w3 for given two weights w1, w2 is based on minimizing the expected loss over a uniform distribution on the curve with respect to w3. Investigating the test accuracy of ensembles of w1 and w(t) on the curve by changing t suggests that even the ensemble of w1 and w(t) close to w1 have better accuracy than each of them. So by perturbing w1 around w1 by gradient descent with cyclical learning rate scheduling, we can collect weights at lower learning rate points, and the ensemble of these weights (FGE) have much better accuracy. The experiments also provide enough evidences that this idea work using a set of the-state-of-the-art architectures such as ResNet-110 and WRN-28-10 as well as more standard VGG-16.  Strength: - The finding that the loss surfaces have the connecting curves between any converged weights w1 and w2 where the training and test accuracy is nearly constant is very interesting. This phenomena is presented clearly with some illustrative examples for the modern CNN architectures such as ResNet and WRN.   - It is also interesting to confirm how far we need to move along a connecting curve to find a point nice to be mixed into the ensemble. This well motivates a proposed ensembling method FGE.   - Compared to the other ensembling such as Snapshot Ensembles [9], the proposed FGE would be nicely presented, and shows good empirical performance.  Weakness: - The uniqueness of connecting curves between two weights would be unclear, and there might be a gap between the curve and FGE. A natural question would be, for example, if we run the curve findings several times, we will see many different curves? Or, those curves would be nearly unique?   - The evidences are basically empirical, and it would be nice if we have some supportive explanations on why this curve happens (and whether it always happens).  - The connections of the curve finding (the first part) and FGE (the second part) would be rather weak. When I read the first part and the title, I imagined that take random weights, learn curves between weights, and find nice wights to be mixed into the final ensemble, but it was not like that. (this can work, but also computationally demanding)   Comment: - Overall I liked the paper even though the evidences are empirical. It was fun to read. The reported phenomena are quite mysterious, and interesting enough to inspire some subsequent research.  - To be honest, I'm not sure the first curve-finding part explains well why the FGE work. The cyclical learning rate scheduling would perturb the weight around the initial converged weight, but it cannot guarantee that weight is changing along the curve described in the first part.