Training Neural Networks for and by Interpolation
In modern supervised learning, many deep neural networks are able to interpolate the data: the empirical loss can be driven to near zero on all samples simultaneously. In this work, we explicitly exploit this interpolation property for the design of a new optimization algorithm for deep learning. Specifically, we use it to compute an adaptive learning-rate in closed form at each iteration. This results in the Adaptive Learning-rates for Interpolation with Gradients (ALI-G) algorithm. ALI-G retains the main advantage of SGD which is a low computational cost per iteration. But unlike SGD, the learning-rate of ALI-G uses a single constant hyper-parameter and does not require a decay schedule, which makes it considerably easier to tune. We provide convergence guarantees of ALI-G in the stochastic convex setting. Notably, all our convergence results tackle the realistic case where the interpolation property is satisfied up to some tolerance. We provide experiments on a variety of architectures and tasks: (i) learning a differentiable neural computer; (ii) training a wide residual network on the SVHN data set; (iii) training a Bi-LSTM on the SNLI data set; and (iv) training wide residual networks and densely connected networks on the CIFAR data sets. ALI-G produces state-of-the-art results among adaptive methods, and even yields comparable performance with SGD, which requires manually tuned learning-rate schedules. Furthermore, ALI-G is simple to implement in any standard deep learning framework and can be used as a drop-in replacement in existing code.
Thanks for the responses and my concerns seem to be addressed. But since I do not know much about this area, I would like to stick to my initial rating 6. ================================================================================================== This work designs a new optimization SGD algorithm named ALI-G for deep neural network with interpolation property. This algorithm only has a single hyper-parameter and doesn’t have a decay schedule. The authors provide the convergence guarantees of ALI-G in the stochastic convex setting as well as the experiment results on four tasks.This paper shows state-of-the-art results but I still have have two concerns. My main concern is that the performances of other SGD algorithms may be potentially better than the results showed in section 5 because it is not easy to tune the parameter. It would be better if the authors can tune the hyper-parameter more carefully. Take figure 3 as an example. The settings where step size bigger than 1e+1 can hardly shows something, because the step-size is too big for SGD to converge. The settings where step size smaller 1e-2 can also hardly shows something, because the step-size is too small and the experiments only runs 10k steps. It would be better if authors can do more experiments in the settings where step size is from 1e-3 to 1e+0. Moreover, the optimal step size of different optimization algorithms may differ a lot. It would be much more fair if the authors can compare the best performance of different algorithms. Another concern is that the authors only give the convergence rate of ALI-G in section 3 but haven’t make any comparisons. For example, it would be better if the authors can show that ALI-G has better convergence result than vanilla SGD without decay schedule.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Training Neural Networks for and by Interpolation
In modern supervised learning, many deep neural networks are able to interpolate the data: the empirical loss can be driven to near zero on all samples simultaneously. In this work, we explicitly exploit this interpolation property for the design of a new optimization algorithm for deep learning. Specifically, we use it to compute an adaptive learning-rate in closed form at each iteration. This results in the Adaptive Learning-rates for Interpolation with Gradients (ALI-G) algorithm. ALI-G retains the main advantage of SGD which is a low computational cost per iteration. But unlike SGD, the learning-rate of ALI-G uses a single constant hyper-parameter and does not require a decay schedule, which makes it considerably easier to tune. We provide convergence guarantees of ALI-G in the stochastic convex setting. Notably, all our convergence results tackle the realistic case where the interpolation property is satisfied up to some tolerance. We provide experiments on a variety of architectures and tasks: (i) learning a differentiable neural computer; (ii) training a wide residual network on the SVHN data set; (iii) training a Bi-LSTM on the SNLI data set; and (iv) training wide residual networks and densely connected networks on the CIFAR data sets. ALI-G produces state-of-the-art results among adaptive methods, and even yields comparable performance with SGD, which requires manually tuned learning-rate schedules. Furthermore, ALI-G is simple to implement in any standard deep learning framework and can be used as a drop-in replacement in existing code.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



Thanks for the responses and my concerns seem to be addressed. But since I do not know much about this area, I would like to stick to my initial rating 6. ================================================================================================== This work designs a new optimization SGD algorithm named ALI-G for deep neural network with interpolation property. This algorithm only has a single hyper-parameter and doesn’t have a decay schedule. The authors provide the convergence guarantees of ALI-G in the stochastic convex setting as well as the experiment results on four tasks.This paper shows state-of-the-art results but I still have have two concerns. My main concern is that the performances of other SGD algorithms may be potentially better than the results showed in section 5 because it is not easy to tune the parameter. It would be better if the authors can tune the hyper-parameter more carefully. Take figure 3 as an example. The settings where step size bigger than 1e+1 can hardly shows something, because the step-size is too big for SGD to converge. The settings where step size smaller 1e-2 can also hardly shows something, because the step-size is too small and the experiments only runs 10k steps. It would be better if authors can do more experiments in the settings where step size is from 1e-3 to 1e+0. Moreover, the optimal step size of different optimization algorithms may differ a lot. It would be much more fair if the authors can compare the best performance of different algorithms. Another concern is that the authors only give the convergence rate of ALI-G in section 3 but haven’t make any comparisons. For example, it would be better if the authors can show that ALI-G has better convergence result than vanilla SGD without decay schedule.
assistant
Thanks for the responses and my concerns seem to be addressed. But since I do not know much about this area, I would like to stick to my initial rating 6. ================================================================================================== This work designs a new optimization SGD algorithm named ALI-G for deep neural network with interpolation property. This algorithm only has a single hyper-parameter and doesn’t have a decay schedule. The authors provide the convergence guarantees of ALI-G in the stochastic convex setting as well as the experiment results on four tasks.This paper shows state-of-the-art results but I still have have two concerns. My main concern is that the performances of other SGD algorithms may be potentially better than the results showed in section 5 because it is not easy to tune the parameter. It would be better if the authors can tune the hyper-parameter more carefully. Take figure 3 as an example. The settings where step size bigger than 1e+1 can hardly shows something, because the step-size is too big for SGD to converge. The settings where step size smaller 1e-2 can also hardly shows something, because the step-size is too small and the experiments only runs 10k steps. It would be better if authors can do more experiments in the settings where step size is from 1e-3 to 1e+0. Moreover, the optimal step size of different optimization algorithms may differ a lot. It would be much more fair if the authors can compare the best performance of different algorithms. Another concern is that the authors only give the convergence rate of ALI-G in section 3 but haven’t make any comparisons. For example, it would be better if the authors can show that ALI-G has better convergence result than vanilla SGD without decay schedule.