We study the implicit bias of gradient descent methods in solving a binary classification problem over a linearly separable dataset. The classifier is described by a nonlinear ReLU model and the objective function adopts the exponential loss function. We first characterize the landscape of the loss function and show that there can exist spurious asymptotic local minima besides asymptotic global minima. We then show that gradient descent (GD) can converge to either a global or a local max-margin direction, or may diverge from the desired max-margin direction in a general context. For stochastic gradient descent (SGD), we show that it converges in expectation to either the global or the local max-margin direction if SGD converges. We further explore the implicit bias of these algorithms in learning a multi-neuron network under certain stationary conditions, and show that the learned classifier maximizes the margins of each sample pattern partition under the ReLU activation.
This paper considers the binary classification problem with exponential loss and ReLu activation function (single neuron). The authors characterize the asymptotic loss landscape by three different types of critical points. They prove that gradient descent (GD) will result in four different regions and provide convergence rates for GD to converge to an asymptotic global minimum, asymptotic local minimum and local minimum under certain assumptions. The authors also provide convergence results for stochastic gradient descent (SGD) and provide extensions to leaky ReLu activation and multi-neuron networks. The paper is well written and the results are mostly clearly presented. This paper mostly follows the line of research by Soudry et al. (2017, 2018), while it has its own merit due to the ReLu activation function considered. However, there are many strong assumptions that are not carefully verified and I really have concerns about the contribution of this paper since they simplify their analysis and results merely by imposing stringent conditions. In particular, I have the following major comments about the paper:
1.	In the definition of max-margin direction, why you use \argmin_{w} max_{i} (w^{\top}x_i)? It seems to me that the definition should be \argmax_{w} min_{i} (w^{\top}x_i). This definition keeps appearing in multiple places in the main paper. 
2.	In the proof of Theorem 3.2, I am confused by the argument of the case that \hat w^{+} is not in the linearly separable region. More clarification is needed to make the proof rigorous.
3.	In the analysis of Theorem 3.3 and 3.4, the authors make a very stringent assumption that the iterate w_t staying in linear separable region for all t>\mathcal{T}. This assumption seems too strong, which should be verified rather than imposed in analysis of SGD. Note that even the example shown in Proposition 2 is still very restrictive (you require all the positive examples or negative examples are very close to one another).
4.	Furthermore, in the analysis of SGD, the authors did not specify the assumption that \hat w^{+} lies in the linear separable region, which is also required in this theorem and also very strong. Given such strong assumptions, the analytic results seem to be trivial and it is hard to evaluate the authors’ contribution.  
5.	For the convergence results of SGD, the current rate is derived on the distance between \|E[w_t] - \hat{w}\|^2. Can you provide similar results for mean square error (E\| w_t - \hat{w} \|^2)? 
6.	In multi-neuron case, the authors again make very strong assumptions that all the neurons have unchanging activation status. This is not easily achievable without careful characterization or other rigorous assumptions. Under such strong assumptions, the extension to multi-neuron again seems not very meaningful.
Other minor comments:
1.	The references are not correctly cited. For instance, please correct the use of parenthesis in “… which is different from that in (Soudry et al., 2017, Corollary 8)” and “… hold for various other types of gradient-based algorithms Gunasekar et al. (2018)”.
2.	The sentence “…, which the nature of convergence is different from …” does not read well. Should it be “where” or “of which”?