In this paper, we adopt distributionally robust optimization (DRO) (Ben-Tal et al., 2013) in hope to achieve a better generalization in deep learning tasks. We establish the generalization guarantees and analyze the localized Rademacher complexity for DRO, and conduct experiments to show that DRO obtains a better performance. We reveal the profound connection between SGD and DRO, i.e., selecting a batch can be viewed as choosing a distribution over the training set. From this perspective, we prove that SGD is prone to escape from bad stationary points and small batch SGD outperforms large batch SGD. We give an upper bound for the robust loss when SGD converges and keeps stable. We propose a novel Weighted SGD (WSGD) algorithm framework, which assigns high-variance weights to the data of the current batch. We devise a practical implement of WSGD that can directly optimize the robust loss. We test our algorithm on CIFAR-10 and CIFAR-100, and WSGD achieves significant improvements over the conventional SGD.
This paper motivates performing a “robustified” version of SGD. It attempts to formalize this notion, proves some variants of generalization bounds, and proposes an algorithm that claims to implement such a modified SGD.
The clarity of the paper can be improved. There are several notational and language ambiguities throughout. Most importantly, a couple of paragraphs that are meant to convey key intuitions about the results are very badly written (the one following Theorem 1, the one preceding Theorem 2, the last one in Section 5.1, and the one preceding Theorem 5, more on these later).
Apart from these clarity issues, the significance of the results is weak. This is because although the technical statements seem correct, the leap from them to measurable outcomes (such as actual generalization bounds) are missing. Part of this is due to a lack of a good notion of “true” robust risk. Moreover the algorithmic contribution does not connect well with the suggested theory and the experimental results are modest at best. Here is a more detailed breakdown of my objections.
-	The notion of distributional robust loss is sound, i.e. R(\theta, K). Its empirical variant is also good, \hat{R}(\theta, K). But the notion of robust loss defined in the paper, \hat R_{S,P}(\theta) with the weights on the samples, breaks from this notion. The reason is that in the former case the weights depend on the value of the sample (z) whereas in the latter they depend on the index (i). It is not evident how to map one to the other.
-	This makes the question of what is the “true” robust risk unclear. It is tempting to simply say it is its expectation with respect to a generic sample. This is the view taken in Theorem 3, which offers a kind of generalization bound. But if one looks carefully at this, the location of the expectation and supremum should be swapped. Here is an alternative view: if want to think of the infinite sample limit, then we need to have a sequence of robustness classes P_m, that vary with m (say those that put weight only on a q-fraction of the samples, just like in the suggested WSGD). The “true” robust risk would be the limit of the sup of the empirical risk, this keeps the sup and expectation in the right order. Under the right conditions, this limit would indeed exist. And it is difficult to know, for a given m, how *far* the generic-sample expectation of Theorem 3 is from it. Without this knowledge, it is difficult to interpret Theorem 3 as a generalization bound.
-	Theorem 1 itself is a standard result. The discussion after Theorem 1 is the kind of argument that also explains Fisher information, and can be presented more clearly. I’m not sure whether Theorem 2 exactly proves what the paragraph before it is trying to explain. The fact that SGD converges into the ball seems contrived, since the quantities that we are trying to bound have nothing to do with the optimization method. If the optimum is within the ball (+/- something) then the same result should hold with the step size replaced with the (+/- something). So how does this explain escaping stationary points?
-	If we accept Theorem 3 as a generalization bound, alongside with the Rademacher bounds of Theorem 4, I don’t think the paper treats the balance between the various terms adequately enough. In particular we see that the |P|_\infty term in Theorem 3 has to balance out the (robust) Rademacher bound, and need it to be of the order of (1+RAD_2(P)\sqrt{\log N}/m). For P that puts weight k over 1/k points, |P|_\infty = 1/k. RAD_2(P) is bounded by 1/\sqrt{k}, so it’s negligible next to 1. But the covering number N can grow exponentially with k (when it’s not too large, and for small \epsilon, just by counting arguments). So this seems to say that for a good tradeoff in the bounds will lead to k having to be a growing fraction of m. This intuition, if true, is not presented. Not only that, but it also goes against the suggested approach of choosing some constant fraction of m.
-	Theorem 5 gives a local Rademacher complexity. But again there is a conceptual step missing from this to strong generalization bounds, partly because we are not exactly minimizing the empirical risk within the considered class. Also, the discussion that follows bounding the rad_\infty with |P|_\infty is deficient, because it misses again the fact there are two salient terms that need to balance out.
-	Algorithm 1 (WSGD) needs to specify (q,r) and which G (G_1 or G_2) as inputs too.
-	Most importantly, WSGD does not seem to be minimizing the robust risk at all. First, I’m not really sure what the G_1 variant does. If we were to follow the intuition of Theorem 1, we should be looking at the gradients, not the loss values. As for G_2, by sampling we are in fact replacing the sup over p with an average over P. This can have a significantly different behavior, and we could possibly interpret it as a slightly reduced effective batch size, especially in the case of G_2. In fact, in the experiments, when r is set to 0, this is exactly what is happening! At any rate, it is not clear at all how any of the earlier sections connect with sections 6 or 7.
-	In the experimental section it is not clarified which of the latter two is used (I assume G_2, the randomized one, given the discussion in the end of Section 6.) When the authors write “accuracy improvement”, they should more clearly say “relative decrease in misclassification error”. That’s the only thing that makes sense with the numbers, and if it does in fact  the authors mistakenly say that the 5-15% improvement is for CIFAR 100 and the 5% is for CIFAR 10, it’s the other way around! And the exception (least) improvement seems to be ResNet-34 on CIFAR-100 (not VGG-16, as they claim, unless the table is wrong.) All in all, these are all pretty weak results, albeit consistent. A better benchmark would have been to compare against various batch sizes, and somehow show that the results do *not* follow from batch size effects.