NEURAL PERSISTENCE: A COMPLEXITY MEASURE FOR DEEP NEURAL NETWORKS USING ALGEBRAIC TOPOLOGY
While many approaches to make neural networks more fathomable have been proposed, they are restricted to interrogating the network with input data. Measures for characterizing and monitoring structural properties, however, have not been developed. In this work, we propose neural persistence, a complexity measure for neural network architectures based on topological data analysis on weighted stratified graphs. To demonstrate the usefulness of our approach, we show that neural persistence reflects best practices developed in the deep learning community such as dropout and batch normalization. Moreover, we derive a neural persistencebased stopping criterion that shortens the training process while achieving comparable accuracies as early stopping based on validation loss.
The authors, motivated by work in topological graph analysis, introduce a new broadly applicable complexity measure they call  neural persistence--essentially a sum over norms of persistence diagrams (objects from the study of persistent homology).  The also provide experiments testing their parameter, primarily on MNIST with some work on CIFAR-10.
I'd like to preface my criticism with the following: this work is extremely compelling, and the results and experiments are sound.  I'm very interested to see where this goes.  Figure 2 is particularly compelling!
That said, I am extremely suspicious of proposals for measures of generalization which (1) do not make contact with the data distribution being studied, and (2) which are only tested on MNIST and CIFAR-10.  Additionally, (3) it is not clear what a "good" neural persistence is, a priori, and (4) I'm not entirely sure I agree with the author's assessment of their numerical data.
In more detail below:
1. At this point, there's a tremendous number of different suggested ways to measure "generalization" by applying different norms and bounds and measures from all of the far reaches of mathematics.  A new proposed measure **really needs** to demonstrate a clear competitive measure against other candidates.  The authors make a strong case that this measure is better than competitors from TGA, but I'm not yet convinced this measure is doing enough legwork.  For example, is it possible that a network has high neural persistence, but still has terrible test or validation error?  Why or why not?  Are there obvious counterexamples?  Are there reasons to think those obvious counterexamples aren't like trained neural networks?  These are all crucial questions to ask and answer if you want this sort of measure to be taken seriously.
2.  Most of your numerical experiments were on MNIST, and MNIST is weird.  It's getting to be a joke now in the community that your idea works on MNIST, but breaks once you try to push it to something harder.  Even Cifar-10 has its quirks, and observations that are true of some networks absolutely do not generalize to others.
3. While I'm convinced that neural persistence allows you to distinguish between networks trained in different ways, it isn't clear why I should expect a particular neural persistence to mean anything at all w.r.t. validation loss.  Are there situations in which the neural persistence has stopped changing, but validation loss is still changing appreciably?  Why or why not?
4. I'm concerned that the early stopping procedure used as a benchmark wasn't tuned as carefully as neural persistence was.  I also honestly cannot determine anything from Figure 4 except that your "Fixed" baseline is bad, and that persistence seems to do about the same as validation loss.  It even seems that Training loss is a better early stopping criteria (better than both validation and persistence!) from this plot, because it seems to perform just as well, and systematically stop earlier.  Am I reading this plot right (particularly for 1.0 fraction MNIST)?
This work currently seems like a strong candidate for the workshop track.  I would have difficulty raising my score above much more than a 6 without much more numerical data, and analysis of when the measure fails.
Edit: The authors have made a significant effort to address my concerns, and I'm updating my score to 7 from 5 in response.