Distributionally Robust Neural Networks
Overparameterized neural networks trained to minimize average loss can be highly accurate on average on an i.i.d. test set, yet consistently fail on atypical groups of the data (e.g., by learning spurious correlations that do not hold at test time). Distributionally robust optimization (DRO) provides an approach for learning models that instead minimize worst-case training loss over a set of pre-defined groups. We find, however, that naively applying DRO to overparameterized neural networks fails: these models can perfectly fit the training data, and any model with vanishing average training loss will also already have vanishing worst-case training loss. Instead, the poor worst-case performance of these models arises from poor generalization on some groups. As a solution, we show that increased regularization---e.g., stronger-than-typical weight decay or early stopping---allows DRO models to achieve substantially higher worst-group accuracies, with 10% to 40% improvements over standard models on a natural language inference task and two image tasks, while maintaining high average accuracies. Our results suggest that regularization is critical for worst-group performance in the overparameterized regime, even if it is not needed for average performance. Finally, we introduce and provide convergence guarantees for a stochastic optimizer for this group DRO setting, underpinning the empirical study above.
To the best of my knowledge, this is the first paper to carefully address and propose an algorithm (with guarantees) for distributionally robust learning in the overparametrized regime, which is typical of modern large deep neural networks. Following other work, the paper formalizes distributionally robust learning as the minimization of a worst-case loss over a set of possible distributions. The main message of the paper is that for distributionally robust learning, regularization plays an important role by avoiding perfect fitting to the training data, at the cost of poor generalization (thus lack of robustness) in some of the possible distributions. Furthermore, the paper proposes a new stochastic optimization algorithm to minimize the loss that corresponds to distributionally robust learning and gives convergence guarantees for the proposed algorithm. I think this is an interesting and solid paper, with a clear presentation style, and well-supported contributions. To the best of my knowledge, this is novel work and, in my opinion, it is relevant work, both in terms of applicability as well as in terms of contribution to the understanding of the generalization behavior of overparameterized deep neural networks.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Distributionally Robust Neural Networks
Overparameterized neural networks trained to minimize average loss can be highly accurate on average on an i.i.d. test set, yet consistently fail on atypical groups of the data (e.g., by learning spurious correlations that do not hold at test time). Distributionally robust optimization (DRO) provides an approach for learning models that instead minimize worst-case training loss over a set of pre-defined groups. We find, however, that naively applying DRO to overparameterized neural networks fails: these models can perfectly fit the training data, and any model with vanishing average training loss will also already have vanishing worst-case training loss. Instead, the poor worst-case performance of these models arises from poor generalization on some groups. As a solution, we show that increased regularization---e.g., stronger-than-typical weight decay or early stopping---allows DRO models to achieve substantially higher worst-group accuracies, with 10% to 40% improvements over standard models on a natural language inference task and two image tasks, while maintaining high average accuracies. Our results suggest that regularization is critical for worst-group performance in the overparameterized regime, even if it is not needed for average performance. Finally, we introduce and provide convergence guarantees for a stochastic optimizer for this group DRO setting, underpinning the empirical study above.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



To the best of my knowledge, this is the first paper to carefully address and propose an algorithm (with guarantees) for distributionally robust learning in the overparametrized regime, which is typical of modern large deep neural networks. Following other work, the paper formalizes distributionally robust learning as the minimization of a worst-case loss over a set of possible distributions. The main message of the paper is that for distributionally robust learning, regularization plays an important role by avoiding perfect fitting to the training data, at the cost of poor generalization (thus lack of robustness) in some of the possible distributions. Furthermore, the paper proposes a new stochastic optimization algorithm to minimize the loss that corresponds to distributionally robust learning and gives convergence guarantees for the proposed algorithm. I think this is an interesting and solid paper, with a clear presentation style, and well-supported contributions. To the best of my knowledge, this is novel work and, in my opinion, it is relevant work, both in terms of applicability as well as in terms of contribution to the understanding of the generalization behavior of overparameterized deep neural networks.
assistant
To the best of my knowledge, this is the first paper to carefully address and propose an algorithm (with guarantees) for distributionally robust learning in the overparametrized regime, which is typical of modern large deep neural networks. Following other work, the paper formalizes distributionally robust learning as the minimization of a worst-case loss over a set of possible distributions. The main message of the paper is that for distributionally robust learning, regularization plays an important role by avoiding perfect fitting to the training data, at the cost of poor generalization (thus lack of robustness) in some of the possible distributions. Furthermore, the paper proposes a new stochastic optimization algorithm to minimize the loss that corresponds to distributionally robust learning and gives convergence guarantees for the proposed algorithm. I think this is an interesting and solid paper, with a clear presentation style, and well-supported contributions. To the best of my knowledge, this is novel work and, in my opinion, it is relevant work, both in terms of applicability as well as in terms of contribution to the understanding of the generalization behavior of overparameterized deep neural networks.