Hi @markdaoust,
Thanks so much for taking a look at this example!
Here is my intuition behind this and why it works in the first place after reading the paper especially for this example.
As I understand GC calculates the mean of the column matrices in our gradient matrix and subtract this mean from each column matrices to have zero mean because unlike the first thing we probably thought, normalizing gradients, does not work very well. About how I understand this works, we could say that in an intuitive way (without notation):
weights (with GC) = gradient of L with respect to W matrix (the standard term) - mean x (gradient of L with respect to W matrix)
We can see that our modified weights now can be seen as the projected gradients along; a unit vector with the number of columns in the weight matrix as the dimension. And similar to the intuition behind batch normalization; this would constrict the weight to a hyperplane. So, this would allow us to regularize the weight space and improve the generalization capability specially when there are less examples (like shown in the example). And this would work across all kind of optimizers, right?
The paper also talks about regularizing output feature space, if we use GC to update weights for SGD based optimizers, for a feature x and x + some scalar, the paper derives that change of output activation caused is only dependent on the scalar and the mean of initial weights, not the final weights. So if the mean for initial weights is very close to 0, we end up with making the output feature space more robust to variations in training data. This would also work on complex optimizers derived from SGD.
Apart of this in my opinion another major aspect showed by the original paper in section 4.2 1 is when they try to compare original loss and the constrained loss to show how the optimization can be smoothened out reducing training time, howveer they straight up derive this with Lipschitzness, unlike others for which I shared my understanding of them geometrically.