Most neural networks are trained using first-order optimization methods, which are sensitive to the parameterization of the model. Natural gradient descent is invariant to smooth reparameterizations because it is defined in a coordinate-free way, but tractable approximations are typically defined in terms of coordinate systems, and hence may lose the invariance properties. We analyze the invariance properties of the Kronecker-Factored Approximate Curvature (K-FAC) algorithm by constructing the algorithm in a coordinate-free way. We explicitly construct a Riemannian metric under which the natural gradient matches the K-FAC update; invariance to affine transformations of the activations follows immediately. We extend our framework to analyze the invariance properties of K-FAC appied to convolutional networks and recurrent neural networks, as well as metrics other than the usual Fisher metric.
This paper is concerned with tractable (approximate) forms of natural gradient updates for neural networks, in particular with the recent K-FAC approximation, which applies a set of approximation (layer-wise independence, Kronecker structure for affine maps) in order to obtain a Hessian that can be computed and inverted efficiently. K-FAC has been introduced for MLPs, and has previously been generalized to convolutional and certain recurrent NNs.
The stated goal of this paper is to provide a mathematical re-formulation of K-FAC in terms of Riemannian metrics. While K-FAC has been developed as approximation to the exact natural gradient update, they come up with a different Riemannian metric, definition of space, etc., such that in the end, K-FAC is the exact natural gradient for that. The authors here also obtain a more precise answer to invariance properties and, given some heavy maths, what they claim to be more elegant proofs of previously known properties of K-FAC.
The paper uses very heavy math, well "over my head" and likely most ICLR attendees. Along with me, they'll ask the obvious question of what this is good for. As far as I can see, there is nothing really new being proposed here in terms of practical consequences. The authors also do not make much effort to explain why their viewpoint is useful, say to obtain practically relevant insights in future work. So, as far as I am concerned, I do not see why this work should be of much relevance to ICLR, which is not an abstract maths conference.
A final comment is that people have for a very long time tried to use second-order optimization for MLPs. The aspect that always was tricky there, is that SGD is *stochastic*, and the second-order info is hard to estimate from a mini-batch. The sets of approximations of K-FAC are pretty extreme, but they may just be needed to make things work in the end, because they may stabilize that "stochastic inverse Fisher info matrix" enough to not make the optimization process fail altogether. Now, all theoretical arguments, like "invariance to this and that", always ignore the crucial fact that you are talking about a stochastic estimate over a mini-batch, and your theory is always for E_x[...] "being the truth". It is not, it is just over a small mini-batch. I am not saying the additional theoretical insight from this work here (over previous K-FAC work), whatever it may be in the end, is not useful. I am just saying I'd be a lot more confident if the authors would specifically address the stochastic property.