The goal of the paper is to propose an algorithm for learning the most generalizable solution from given training data. It is shown that Bayesian approach leads to a solution that dependent on statistics of training data and not on particular samples. The solution is stable under perturbations of training data because it is defined by an integral contribution of multiple maxima of the likelihood and not by a single global maximum. Specifically, the Bayesian probability distribution of parameters (weights) of a probabilistic model given by a neural network is estimated via recurrent variational approximations. Derived recurrent update rules correspond to SGD-type rules for finding a minimum of an effective loss that is an average of an original negative log-likelihood over the Gaussian distributions of weights, which makes it a function of means and variances. The effective loss is convex for large variances and non-convex in the limit of small variances. Among stationary solutions of the update rules there are trivial solutions with zero variances at local minima of the original loss and a single non-trivial solution with finite variances that is a critical point at the end of convexity of the effective loss in the mean-variance space. At the critical point both first-and second-order gradients of the effective loss w.r.t. means are zero. The empirical study confirms that the critical point represents the most generalizable solution. While the location of the critical point in the weight space depends on specifics of the used probabilistic model some properties at the critical point are universal and model independent.
Summary: Non-convex learning problems can have multiple solutions with different generalization properties, thus it is important to find solutions that generalize well. The goal of this paper is to derive an algorithm for finding a solution to the learning problem with the best possible generalization properties. This is achieved by using a Bayesian approach in which the parameters w (e.g., weights of a network) are random variables and the effective loss (integral wrt to w) is minimized in lieu of the usual loss. The paper assumes that each component of the weight vector w is Gaussian and derives a formula for updating the mean and covariance of said Gaussians (this is an SGD method). The paper claims that the resulting effective loss is convex for large variances (sigma > threshold), nonconvex for small variances (sigma < threshold), and converges to original loss as sigma goes to zero. The paper also claims that when sigma=0 there are trivial solutions that are unstable as data changes, but that when sigma=threshold (assuming this is what is meant by end of convexity) there are non-trivial solutions that are less sensitive to data changes and hence the most generalizable.
Comments: the goal of the paper (finding minima that generalize well) is an excellent one. But the paper is not clearly written and appears to oversell the contribution. In particular, the title speaks about SGD, dropout, generalization and critical points “at the end of convexity”. Naturally, a reader is inclined to think that the paper will study SGD and dropout for deep learning and analyze generalization properties of the solutions found by those methods. In reality, there is very little in the paper about SGD, dropout, and generalization. The connection with SGD is merely because the method for updating mu and sigma is an SGD method. The connection with dropout is mentioned in passing in one paragraph and it is not very clear. The connection with generalization is claimed but never quite explained (there are no generalization bounds in the paper). As far as understand, the paper considers the minimization of the effective loss, uses a Gaussian approximation for computing the effective loss, and focuses primarily on the characterization of convexity as a function of sigma as well as a characterization of the critical points depending on whether the effective loss is convex (sigma above a threshold) or not (sigma below the threshold). The main claim appears to be that critical points at the critical threshold lead to solutions that generalize well, but a detailed explanation of why this is the case isn't given. If my digest of the paper is the correct one, then modifying the title, abstract and intro to make this clear would have helped a lot. 
Beyond the high-level lack of clarity about the contribution of the paper, the writing lacks precision and rigor, and many things are undefined (though one can figure them out after reading many times back and forth). Specifically: 
1) It is not explained why the probability of each training sample can be expressed as a product of factors close to 1, with the product taken over the epochs.
2) It is not explained why each factor can be modeled as a product of Gaussians
3) At nearly the top of page 3, a product over n is substituted by a product over t, with x_n replaced by x_t and so forth, but the total number of products goes up from N to TxN. What is the value of y_{NT} and x_{NxT}? Do the authors mean that y_n should have been replaced by y_{n,t} and we now have two indices? Or do the authors mean that the same mini batch of N samples is reused, and so indices should be corrected accordingly? 
4) It is not clear why replacing R_t/Q_{t+1} by 1 is an adequate approximation.
5) At the top of equation 4, there is a product, but no index wrt which the product is taken. Right after it says the index is t, but there is no t in the expression. Should mu_0 be mu_t and similarly for sigma? 
In short, a promising direction, but the contribution of the paper appears to be over claimed and the writing of the paper needs significant improvement before the paper can be accepted for publication.