Convergence Properties of Deep Neural Networks on Separable Data
While a lot of progress has been made in recent years, the dynamics of learning in deep nonlinear neural networks remain to this day largely misunderstood. In this work, we study the case of binary classification and prove various properties of learning in such networks under strong assumptions such as linear separability of the data. Extending existing results from the linear case, we confirm empirical observations by proving that the classification error also follows a sigmoidal shape in nonlinear architectures. We show that given proper initialization, learning expounds parallel independent modes and that certain regions of parameter space might lead to failed training. We also demonstrate that input norm and features* frequency in the dataset lead to distinct convergence speeds which might shed some light on the generalization capabilities of deep neural networks. We provide a comparison between the dynamics of learning with cross-entropy and hinge losses, which could prove useful to understand recent progress in the training of generative adversarial networks. Finally, we identify a phenomenon that we baptize gradient starvation where the most frequent features in a dataset prevent the learning of other less frequent but equally informative features.
 The underlying motivation for the paper is really interesting and cuts straight to the heart of Deep Learning and strives to unravel the key understanding that we are still to a large extent missing. When it comes to clarity and organization I find the paper a bit "messy" in that it is a collection of quite a few findings on the very specific topic of binary classification with quite strong assumptions. Especially given the very specific nature of the topic I miss a strong and clear path through the paper. Unfortunately the paper leaves me with the distinct feeling that there are still a lot of work needed to be able to tell the story about the problem under study. Having said that the paper does contain several individual findings. Having said that I find the ideas leading up to what the authors refers to as "gradient starvation" to be really interesting and that would be a great clear idea to focus on. A few concrete questions/comments: Can you explain somewhere exactly what you mean when you say "learning dynamics of deep learning"? Given the specific nature of the results presented in the paper it would be nice to be precise also when it comes to the overall topic under study. Given the very specific nature of the topic treated in the paper I find the title of the paper largely misleading. The title claims way more than what is actually delivered in the paper, despite the fact that the authors have put in an "On" in the beginning of the title. In Corollary 3.3. you characterize the convergence speed in a nice way, but I am missing the link to the behaviors observed empirically in e.g. Fig. 2. What am I missing? The final sentence in Section 2 is highly speculative and I find this hard to believe without solid backing. The sentence reads "... and helps develop intuitions about behaviors observed in more general settings." Given the restrictive nature of your set-up I find it very hard to believe that this extends to more general settings. Tiny detail: The axes of several of the plots given in the paper mis the lables which makes it hard to read. Straightforward to fix, but worth mentioning nevertheless.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Convergence Properties of Deep Neural Networks on Separable Data
While a lot of progress has been made in recent years, the dynamics of learning in deep nonlinear neural networks remain to this day largely misunderstood. In this work, we study the case of binary classification and prove various properties of learning in such networks under strong assumptions such as linear separability of the data. Extending existing results from the linear case, we confirm empirical observations by proving that the classification error also follows a sigmoidal shape in nonlinear architectures. We show that given proper initialization, learning expounds parallel independent modes and that certain regions of parameter space might lead to failed training. We also demonstrate that input norm and features* frequency in the dataset lead to distinct convergence speeds which might shed some light on the generalization capabilities of deep neural networks. We provide a comparison between the dynamics of learning with cross-entropy and hinge losses, which could prove useful to understand recent progress in the training of generative adversarial networks. Finally, we identify a phenomenon that we baptize gradient starvation where the most frequent features in a dataset prevent the learning of other less frequent but equally informative features.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 The underlying motivation for the paper is really interesting and cuts straight to the heart of Deep Learning and strives to unravel the key understanding that we are still to a large extent missing. When it comes to clarity and organization I find the paper a bit "messy" in that it is a collection of quite a few findings on the very specific topic of binary classification with quite strong assumptions. Especially given the very specific nature of the topic I miss a strong and clear path through the paper. Unfortunately the paper leaves me with the distinct feeling that there are still a lot of work needed to be able to tell the story about the problem under study. Having said that the paper does contain several individual findings. Having said that I find the ideas leading up to what the authors refers to as "gradient starvation" to be really interesting and that would be a great clear idea to focus on. A few concrete questions/comments: Can you explain somewhere exactly what you mean when you say "learning dynamics of deep learning"? Given the specific nature of the results presented in the paper it would be nice to be precise also when it comes to the overall topic under study. Given the very specific nature of the topic treated in the paper I find the title of the paper largely misleading. The title claims way more than what is actually delivered in the paper, despite the fact that the authors have put in an "On" in the beginning of the title. In Corollary 3.3. you characterize the convergence speed in a nice way, but I am missing the link to the behaviors observed empirically in e.g. Fig. 2. What am I missing? The final sentence in Section 2 is highly speculative and I find this hard to believe without solid backing. The sentence reads "... and helps develop intuitions about behaviors observed in more general settings." Given the restrictive nature of your set-up I find it very hard to believe that this extends to more general settings. Tiny detail: The axes of several of the plots given in the paper mis the lables which makes it hard to read. Straightforward to fix, but worth mentioning nevertheless.
assistant
 The underlying motivation for the paper is really interesting and cuts straight to the heart of Deep Learning and strives to unravel the key understanding that we are still to a large extent missing. When it comes to clarity and organization I find the paper a bit "messy" in that it is a collection of quite a few findings on the very specific topic of binary classification with quite strong assumptions. Especially given the very specific nature of the topic I miss a strong and clear path through the paper. Unfortunately the paper leaves me with the distinct feeling that there are still a lot of work needed to be able to tell the story about the problem under study. Having said that the paper does contain several individual findings. Having said that I find the ideas leading up to what the authors refers to as "gradient starvation" to be really interesting and that would be a great clear idea to focus on. A few concrete questions/comments: Can you explain somewhere exactly what you mean when you say "learning dynamics of deep learning"? Given the specific nature of the results presented in the paper it would be nice to be precise also when it comes to the overall topic under study. Given the very specific nature of the topic treated in the paper I find the title of the paper largely misleading. The title claims way more than what is actually delivered in the paper, despite the fact that the authors have put in an "On" in the beginning of the title. In Corollary 3.3. you characterize the convergence speed in a nice way, but I am missing the link to the behaviors observed empirically in e.g. Fig. 2. What am I missing? The final sentence in Section 2 is highly speculative and I find this hard to believe without solid backing. The sentence reads "... and helps develop intuitions about behaviors observed in more general settings." Given the restrictive nature of your set-up I find it very hard to believe that this extends to more general settings. Tiny detail: The axes of several of the plots given in the paper mis the lables which makes it hard to read. Straightforward to fix, but worth mentioning nevertheless.