MEASURING AND REGULARIZING NETWORKS IN FUNCTION SPACE
To optimize a neural network one often thinks of optimizing its parameters, but it is ultimately a matter of optimizing the function that maps inputs to outputs. Since a change in the parameters might serve as a poor proxy for the change in the function, it is of some concern that primacy is given to parameters but that the correspondence has not been tested. Here, we show that it is simple and computationally feasible to calculate distances between functions in a L 2 Hilbert space. We examine how typical networks behave in this space, and compare how parameter 2 distances compare to function L 2 distances between various points of an optimization trajectory. We find that the two distances are nontrivially related. In particular, the L 2 / 2 ratio decreases throughout optimization, reaching a steady value around when test error plateaus. We then investigate how the L 2 distance could be applied directly to optimization. We first propose that in multitask learning, one can avoid catastrophic forgetting by directly limiting how much the input/output function changes between tasks. Secondly, we propose a new learning rule that constrains the distance a network can travel through L 2 -space in any one update. This allows new examples to be learned in a way that minimally interferes with what has previously been learned. These applications demonstrate how one can measure and regularize function distances directly, without relying on parameters or local approximations like loss curvature.
This paper proposes a method for functional regularization for training neural nets, such that the sequence of neural nets during training is stable in function space. Specifically, the authors define a L2 norm (i.e., a Hilbert norm), which can be used to measure distances in this space between two functions. The authors argue that this can aid in preventing catastrophic forgetting, which is demonstrated in a synthetic multi-task variant of MNIST.   The authors also show how to regularize the gradient updates to be conservative in function space in standard stochastic gradient style learning, but with rather inconclusive empirical results.  The authors also draw upon a connection to the natural gradient.
***Clarity***
The paper is reasonably well written.  I think the logical flow could be improved at places.   I think the major issue with clarity is the title.  The authors use the term "regularizing" in a fairly narrow sense, in particular regularizing the training trajectory to be stable in function space.  However, the more dominant usage for regularizing is to regularize the final learned function to some prior, which is not studied or even really discussed in the paper.
Detailed comments:
-- The notation in Section 2 could be cleaned up.  The use of \mu is a bit disconnected from the rest of the notation.  
-- Computing the empirical L2 distance accurately can also be NP hard.  There's no stated guarantee of how large N needs to be to have a good empirical estimate.  Figure 3 is nice, but I think a more thorough discussion on this point could be useful.
-- L2-Space was never formally defined.  
-- Section 2.1 isn't explained clearly.  For instance, in the last paragraph, the first sentence states "the networks are initialized at very different point", and halfway into the paragraph a sentence states "all three initializations begin at approximately the same point in function space.".  The upshot is that Figure 1 doesn't crisply capture the intuition the authors aim to convey.
***Originality***
Strictly speaking, the proposed formulation is novel as far as I am aware.  However, the basic idea has been the air for a while.  For instance, there are some related work in RL/IL on functional regularization:
-- https://arxiv.org/abs/1606.00968
The proposed formulation is, in some sense, the obvious thing to try (which is a good thing).  The detailed connection to the natural gradient is nice.  I do wish that the authors made stronger use of properties of a Hilbert space, as the usage of Hilbert spaces is fairly superficial.  For instance, one can apply operators in a Hilbert space, or utilize an inner product.  It just feels like there was a lost opportunity to really explore the implications.
***Significance***
This is the place where the contributions of this paper are most questionable.  While the multi-task MNIST experiments are nice in demonstrating resilience against catastrophic forgetting, the experiments are pretty synthetic.  What about a more "real" multi-task learning problem?
More broadly, it feels like this paper is suffering from a bit of an identity crisis.  It uses regularizing in a narrow sense to generate conservative updates.  It argues that this can help in catastrophic forgetting.  It also shows how to employ this to construct the standard bounded-update gradient descent rules, although without much rigorous discussion for the implications.  There are some nice empirical results on a synthetic multi-task learning task, and inconclusive results otherwise.  There's a nice little discussion on the connection to the natural gradient.  It argues that that this form of regularization lives in a Hilbert space, but the usage of a Hilbert space is fairly superficial.  All in all, there are some nice pieces of work here and there, but it's all together neither here or there in terms of an overall contribution.    
***Overall Quality***
I think if the authors really pushed one of the angles to a more meaningful contribution, this paper would've been much stronger.  As it stands, the paper just feels too scattered in its focus, without a truly compelling result, either theoretically or empirically.