How regularization affects the critical points in linear networks
This paper is concerned with the problem of representing and learning a linear transformation using a linear neural network. In recent years, there is a growing interest in the study of such networks, in part due to the successes of deep learning. The main question of this body of research (and also of our paper) is related to the existence and optimality properties of the critical points of the mean-squared loss function. An additional primary concern of our paper pertains to the robustness of these critical points in the face of (a small amount of) regularization. An optimal control model is introduced for this purpose and a learning algorithm (backprop with weight decay) derived for the same using the Hamilton's formulation of optimal control. The formulation is used to provide a complete characterization of the critical points in terms of the solutions of a nonlinear matrix-valued equation, referred to as the characteristic equation. Analytical and numerical tools from bifurcation theory are used to compute the critical points via the solutions of the characteristic equation.
The authors of this article consider an infinite-dimensional version of a deep linear network. They propose to learn the weights using an objective function that is the sum of a square loss and a Tikhonov regularizer, controlled by a regularization parameter lambda. The authors derive analytic characterizations for the critical points of the objective, and use them to numerically show that, even for arbitrarily small lambda, local minima and saddle points can appear, that do not exist when there is no regularization.
I am not an expert of this subject, but it seems to me that looking at deep linear networks in an infinite-dimensional setting, instead of the usual finite-dimensional one, is a good idea. It seems to make computations simpler or at least more elegant, and could then allow for more in-depth analysis.
Using methods from optimal control to study this problem is original, and apparently efficient.
I also found the article well-written: it was pleasant to read, despite the fact that I am very unfamiliar with optimal control.
However, I am not sure that the picture that the authors give of the critical points in the limit where lambda goes to zero is clear enough so that I understand what is surprising in it, and what algorithmic implications it can have.
Indeed, when lambda is exactly zero, there are a lot of global minima. If these global minima can be grouped into distinct connected components, then, when lambda becomes positive, it is "normal" that local minima appear (intuitively, each connected component will typically give rise to at least one local minimum, but only one connected component - the one containing the global solution that minimizes the regularizer - will give rise to a local minimum that is also global). It seems to me that this is what the authors observe, in terms of local minima.
The appearing of saddle points is maybe more surprising, but since they seem to come in "from infinity", they might not be an issue for optimization, at least for small values of lambda.
Also, the authors focus on normal solutions. They are easier to compute than non-normal ones, but they are a priori not the only critical points (at least not when lambda=0). So maybe looking only at the normal critical points yields a very different "critical point diagram" than looking at all the critical points. If not, can the authors explain why?
Finally, this is less important but, after all the formal computations (up to Theorem 2), I was slightly disappointed that the appearing of local minima and saddle points was only established for precise examples, and through numerical computations, and not via more rigorous arguments.
Minor remarks:
- It seems to me that the regularization proposed by the authors is essentially "weight decay", that is well-known among the people that train deep networks; it should be said.
- Proposition 1: it is to me that it is not explicitly said that it only holds for positive values of lambda.
- The covariance is defined as \Sigma_0 in the introduction but subsequently denoted by \Sigma.
- l.129: it is said that the "critical points" will be characterized, but the proof uses Proposition 1, that deals with "minimizers", and not "critical points".
- l. 141: "T" is missing in the characteristic equation.
- Theorem 2 - (i): plural should probably be used, since there are several solutions.
- Proof of theorem 2: the proof sketch seems to suggest that the authors assume with no proof that there always exists a normal solution when lambda > 0. The appendix actually proves the existence, but maybe the sketch could be slightly reformulated, so as not to give a wrong impression?
- It seems to me that the authors never show that values of C that satisfy Equation (18) are critical points of (4) (the proposition says the opposite: critical points of (4) satisfy (18)). So how can they be sure that the solutions they compute by analytic continuation are indeed critical points of (4)?
- I am not sure I understand the meaning of "robust" in the conclusion.
Added after author feedback: thank you for the additional explanations. I am however slightly disappointed with your response to my main two comments.
I agree that the emergence of saddle points, as well as the normal / non-normal difference are somewhat surprising. However, it seems to me that it is not really what the article presents as a surprise: if you think that these two points are the most surprising contribution of the article, it should probably be better highlighted and discussed.
For the normal vs non-normal solutions, I do understand that studying non-normal critical points looks very difficult, and I am fine with it. However, I think that the abstract and introduction should then be clear about it, and not talk about general critical points, while the article only describes normal ones.