Learning rules for neural networks necessarily include some form of regularization. Most regularization techniques are conceptualized and implemented in the space of parameters. However, it is also possible to regularize in the space of functions. Here, we propose to measure networks in an L 2 Hilbert space, and test a learning rule that regularizes the distance a network can travel through L 2 -space each update. This approach is inspired by the slow movement of gradient descent through parameter space as well as by the natural gradient, which can be derived from a regularization term upon functional change. The resulting learning rule, which we call Hilbert-constrained gradient descent (HCGD), is thus closely related to the natural gradient but regularizes a different and more calculable metric over the space of functions. Experiments show that the HCGD is efficient and leads to considerably better generalization.
I have read comments and rebuttal - i do not have the luxury of time to read in depth the revision.
It seems that the authors have made an effort to accommodate reviewers' comments. I upgraded the rating.
-----------------------------------------------------------------------------------------------------------------------
Summary: The paper considers the use of natural gradients for learning. The added twist is the substitution of the KL divergence with the Wasserstein distance, as proposed in GAN training. The authors suggest that Wasserstein regularization improves generalization over SGD with a little extra cost.
The paper is structured as follows:
1. KL divergence is used as a similarity measure between two distributions.
2. Regularizing the objective with KL div. seems promising, but expensive.
3. We usually approximate the KL div. with its 2nd order approximation - this introduces the Hessian of the KL divergence, known as Fisher information matrix.
4. However, computing and inverting the Fisher information matrix is computationally expensive.
5. One solution is to approximate the solution F^{-1} J using gradient descent. However, still we need to calculate F. There are options where F could be formed as the outer product of a collection gradients of individual examples ('empirical Fisher').
6. This paper does not move towards Fisher information, but towards Wasserstein distance: after a "good" initialization via SGD is obtained, the inner loop continues updating that point using the Wasserstein regularized objective. 
7. No large matrices need to be formed or inverted, however more passes needed per outer step.
Importance:
Somewhat lack of originality and poor experiments lead to low importance.
Clarity:
The paper needs major revision w.r.t. presenting and highlighting the new main points. E.g., one needs to get to page 5 to understand that the paper is just based on the WGAN ideas in Arjovsky et al., but with a different application (not GANS).
Originality/Novelty:
The paper, based on WGAN motivation, proposes Wasserstein distance regularization over KL div. regularization for training of simple models, such as neural networks. Beyond this, the paper does not provide any futher original idea. So, slight to no novelty.
Main comments:
1. Would the approximation of C_0 by its second-order Taylor expansion (that also introduces a Hessian) help? This would require the combination of two Hessian matrices.
2. Experiments are really demotivating: it is not clear whether using plain SGD or the proposed method leads to better results. 
Overall:
Rejection.