Published as a conference paper at ICLR 2018 MEASURING THE INTRINSIC DIMENSION OF OBJECTIVE LANDSCAPES
Many recently trained neural networks employ large numbers of parameters to achieve good performance. One may intuitively use the number of parameters required as a rough gauge of the difficulty of a problem. But how accurate are such notions? How many parameters are really needed? In this paper we attempt to answer this question by training networks not in their native parameter space, but instead in a smaller, randomly oriented subspace. We slowly increase the dimension of this subspace, note at which dimension solutions first appear, and define this to be the intrinsic dimension of the objective landscape. The approach is simple to implement, computationally tractable, and produces several suggestive conclusions. Many problems have smaller intrinsic dimensions than one might suspect, and the intrinsic dimension for a given dataset varies little across a family of models with vastly different sizes. This latter result has the profound implication that once a parameter space is large enough to solve a problem, extra parameters serve directly to increase the dimensionality of the solution manifold. Intrinsic dimension allows some quantitative comparison of problem difficulty across supervised, reinforcement, and other types of learning where we conclude, for example, that solving the inverted pendulum problem is 100 times easier than classifying digits from MNIST, and playing Atari Pong from pixels is about as hard as classifying CIFAR-10. In addition to providing new cartography of the objective landscapes wandered by parameterized models, the method is a simple technique for constructively obtaining an upper bound on the minimum description length of a solution. A byproduct of this construction is a simple approach for compressing networks, in some cases by more than 100 times.
This paper proposes an empirical measure of the intrinsic dimensionality of a neural network problem. Taking the full dimensionality to be the total number of parameters of the network model, the authors assess intrinsic dimensionality by randomly projecting the network to a domain with fewer parameters (corresponding to a low-dimensional subspace within the original parameter), and then training the original network while restricting the projections of its parameters to lie within this subspace. Performance on this subspace is then evaluated relative to that over the full parameter space (the baseline). As an empirical standard, the authors focus on the subspace dimension that achieves a performance of 90% of the baseline. The authors then test out their measure of intrinsic dimensionality for fully-connected networks and convolutional networks, for several well-known datasets, and draw some interesting conclusions.
Pros:
* This paper continues the recent research trend towards a better characterization of neural networks and their performance. The authors show a good awareness of the recent literature, and to the best of my knowledge, their empirical characterization of the number of latent parameters is original. 
* The characterization of the number of latent variables is an important one, and their measure does perform in a way that one would intuitively expect. For example, as reported by the authors, when training a fully-connected network on the MNIST image dataset, shuffling pixels does not result in a change in their intrinsic dimensionality. For a convolutional network the observed 3-fold rise in intrinsic dimension is explained by the authors as due to the need to accomplish the classification task while respecting the structural constraints of the convnet.
* The proposed measures seem very practical - training on random projections uses far fewer parameters than in the original space (the baseline), and presumably the cost of determining the intrinsic dimensionality would presumably be only a fraction of the cost of this baseline training.
* Except for the occasional typo or grammatical error, the paper is well-written and organized. The issues are clearly identified, for the most part (but see below...).
Cons:
* In the main paper, the authors perform experiments and draw conclusions without taking into account the variability of performance across different random projections. Variance should be taken into account explicitly, in presenting experimental results and in the definition and analysis of the empirical intrinsic dimension itself. How often does a random projection lead to a high-quality solution, and how often does it not?
* The authors are careful to point out that training in restricted subspaces cannot lead to an optimal solution for the full parameter domain unless the subspace intersects the optimal solution region (which in general cannot be guaranteed). In their experiments (FC networks of varying depths and layer widths for the MNIST dataset), between projected and original solutions achieving 90% of baseline performance, they find an order of magnitude gap in the number of parameters needed. This calls into question the validity of random projection as an empirical means of categorizing the intrinsic dimensionality of a neural network.
* The authors then go on to propose that compression of the network be achieved by random projection to a subspace of dimensionality greater than or equal to the intrinsic dimension. However, I don't think that they make a convincing case for this approach. Again, variation is the difficulty: two different projective subspaces of the same dimensionality can lead to solutions that are extremely different in character or quality. How then can we be sure that our compressed network can be reconstituted into a solution of reasonable quality, even when its dimensionality greatly exceeds the intrinsic dimension?
* The authors argue for a relationship between intrinsic dimensionality and the minimum description length (MDL) of their solution, in that the intrinsic dimensionality should serve as an upper bound on the MDL. However they don't formally acknowledge that there is no standard relationship between the number of parameters and the actual number of bits needed to represent the model - it varies from setting to setting, with some parameters potentially requiring many more bits than others. And given this uncertain connection, and given the lack of consideration given to variation in the proposed measure of intrinsic dimensionality, it is hard to accept that "there is some rigor behind" their conclusion that LeNet is better than FC networks for classification on MNIST because its empirical intrinsic dimensionality score is lower.
* The experimental validation of their measure of intrinsic dimension could be made more extensive. In the main paper, they use three image datasets - MNIST, CIFAR-10 and ImageNet. In the supplemental information, they report intrinsic dimensions for reinforcement learning and other training tasks on four other sets.
Overall, I think that this characterization does have the potential to give insights into the performance of neural networks, provided that variation across projections is properly taken into account. For now, more work is needed.
====================================================================================================
Addendum:
The authors have revised their paper to take into account the effect of variation across projections, with results that greatly strengthen their results and provide a much better justification of their approach. I'm satisfied too with their explanations, and how they incorporated them into their revised version. I've adjusted my rating of the paper accordingly.
One point, however: the revisions seem somewhat rushed, due to the many typos and grammatical errors in the updated sections. I would like to encourage the authors to check their manuscript once more, very carefully, before finalizing the paper.
====================================================================================================