L1 and L2 regularizers are critical tools in machine learning due to their ability to simplify solutions. However, imposing strong L1 or L2 regularization with gradient descent method easily fails, and this limits the generalization ability of the underlying neural networks. To understand this phenomenon, we investigate how and why learning fails for strong regularization. Specifically, we examine how gradients change over time for different regularization strength and provide an analysis why the gradients diminish so fast when strong regularization is imposed. We find that there exists a tolerance level of regularization strength, where the learning completely fails if the regularization strength goes beyond it. We propose a simple but novel method, Delayed Strong Regularization, in order to moderate the tolerance level. Experiment results show that our proposed approach indeed achieves strong regularization for both L1 and L2 regularizers and improves both accuracy and sparsity on public data sets. Our source code is published.
The work was prompted by  an interesting observation: a phase transition can be observed in deep learning with stochastic gradient descent and Tikhonov regularization. When the regularization parameter exceeds a (data-dependent) threshold, the parameters of the model are driven to zero, thereby preventing any learning. The authors then propose to moderate this problem by letting the regularization parameter to be zero for 5 to 10 epochs, and then applying the "strong" penalty parameter. In their experimental results, the phase transition is not observed anymore with their protocol. This leads to better performances, by using penalty parameters that would have prevent learning with the usual protocol.
The problem targeted is important, in the sense that it reveals that some of the difficulties related to non-convexity and the use of SGD that are often overlooked. The proposed protocol is reported to work well, but since it is really ad hoc, it fails to convince the reader that it provides the right solution to the problem. I would have found much more satisfactory to either address the initialization issue by a proper warm-start strategy, or to explore standard optimization tools such as constrained optimization (i.e. Ivanov regularization) , that could be for example implemented by stochastic projected gradient or barrier functions. I think that the problem would be better handled that way than with the proposed strategy, which seems to rely only on a rather limited amount of experiments, and which may prove to be inefficient when dealing with big databases.
To summarize, I believe that the paper addresses an important point, but that the tools advocated are really rudimentary compared with what has been already proposed elsewhere.
Details :
- there is a typo in the definition of the proximal operator in Eq. (9) 
- there are many unsubstantiated speculations in the comments of the experimental section that do not add value to the paper 
- the figure showing the evolution of the magnitude of parameters arrives too late and could be completed by the evolution of the data-fitting term of the training criterion