Sobolev Training for Neural Networks
At the heart of deep learning we aim to use neural networks as function approximators -training them to produce outputs from inputs in emulation of a ground truth function or data creation process. In many cases we only have access to input-output pairs from the ground truth, however it is becoming more common to have access to derivatives of the target output with respect to the input -for example when the ground truth function is itself a neural network such as in network compression or distillation. Generally these target derivatives are not computed, or are ignored. This paper introduces Sobolev Training for neural networks, which is a method for incorporating these target derivatives in addition the to target values while training. By optimising neural networks to not only approximate the function's outputs but also the function's derivatives we encode additional information about the target function within the parameters of the neural network. Thereby we can improve the quality of our predictors, as well as the data-efficiency and generalization capabilities of our learned function approximation. We provide theoretical justifications for such an approach as well as examples of empirical evidence on three distinct domains: regression on classical optimisation datasets, distilling policies of an agent playing Atari, and on large-scale applications of synthetic gradients. In all three domains the use of Sobolev Training, employing target derivatives in addition to target values, results in models with higher accuracy and stronger generalisation.
I have enjoyed reading this paper. The idea is clear and very intuitive. Figure 1 is very well conceived.
Now, I must say that, throughout my reading, I was looking for the authors to "confess" that the first-order derivatives might be entirely useless in certain situations. But in a situation where both f(x) and f'(x) are given, taken as ground truth, then it is certainly a wise idea to use the information about f'(x) in order to train a new network that approximates f(x) to solve a task.
However the problem is that, when a certain model M1 is trained to minimize a loss on a training set, then there is nothing that guarantees the sanity/usefulness of the gradients of M1. The derivatives depend a lot on the activation functions being used, for example. Then I don't see at all why there would be any general principle that would encourage us to imitate those gradients in order to learn another model M2 based on M1.
It is very possible that this doesn't happen in practice, on common datasets and reasonable models, and that's interesting. I think that it's partly what the paper studies : does this thing work in practice ? Okay, great ! Worth discussing in a paper with good experiments. Accept !
In the past, I have had a conversation with one of the great ML authorities in which he/she casually claimed that when two functions (f1, f2) are close, then their derivatives are also close. This doesn't sound unreasonable as a mental shortcut, but it's ABSOLUTELY INCORRECT in the mathematical sense. Yet people use that shortcut.
The authors refer to this indirectly, but I would really like to see the authors be more explicit about that fact, and I think it would make the paper slightly better. Just to be sure that nobody makes that mental shortcut when they read this stimulating paper.
I had a bit more difficulty understanding the subtleties of the Synthetic Gradient portion of the paper. I trust that it's sound.
Also, in Figure 4 I'm looking at certain plots and it seems like "regular distillation" very often gets worse with more iterations. (The x-axis are iterations, right?) Sobolev training looks better, but it's hard to say if it's just because the hyperparameters were not selected correctly. They kinda make regular distillation look bad. Moreover, the y-axis really plays with the scales and makes the differences look way bigger than they are. It's hard to know how meaningful those differences are.
To sum it up, I'm glad that I got to read this paper. I would like to see better, and more meaningful experiments, in a later paper, but I'm happy about what I see now.