Learning Deep Parsimonious Representations
In this paper we aim at facilitating generalization for deep networks while supporting interpretability of the learned representations. Towards this goal, we propose a clustering based regularization that encourages parsimonious representations. Our k-means style objective is easy to optimize and flexible, supporting various forms of clustering, such as sample clustering, spatial clustering, as well as co-clustering. We demonstrate the effectiveness of our approach on the tasks of unsupervised learning, classification, fine grained categorization, and zero-shot learning.
The paper suggests an approach to regularizing neural networks by encouraging their activations to form tight clusters in one of several proposed spaces. A k-means style objective is added to the overall network loss, penalizing deviation of activations in each layer from their closest (layer specific) cluster center. The cluster centers are learned jointly with the network weights.          Three approaches are suggested for clustering: sample clustering, where the vectors to be clustered (at each layer) are the vectorized activations corresponding to different samples in a mini-batch; spatial clustering, where the vectors to be clustered are values across channels corresponding to different locations (and samples) within a convolutional layer's activations (same as sample clustering for fully connected layers); channel co-clustering, where vectors to be clustered correspond to vectorizations of different channels (and samples) within a convolutional layer's activations.          The approaches are demonstrated to have a positive effect on generalization error on multiple datasets and across several tasks, including unsupervised learning on MNIST, classification on CIFAR10 and CIFAR100, and fine-grained classification and zero-shot learning on CUB-200-2011 (birds). Additionally, the clusters obtained provide insight into the structures learned by the network.       
         This is a very solid paper all around. The idea to cluster activations and force the network to "stay close" to a restricted set of representations is intuitive, acting as a parsimony constraint that additionally enables interpretability. While solid theoretical motivation and analysis isn't provided, the proposed algorithm feels natural and the experiments are fairly comprehensive and compelling. A broad range of tasks are considered, including unsupervised, fine-grained, and zero-shot learning in addition to standard classification, and the visualizations of cluster structure demonstrate that the cluster centers are meaningful---they convince me that the approach is performing as one would hope.          Some specific points: It would be nice to see a table or chart demonstrating the effect of different choices of cluster size on generalization error (it is mentioned that cross-validation was used, but it would still be good to get a feel for the sensitivity with respect to choice of cluster size). A concern about the method is of course the added computation in each iteration required to update and utilize the clusters. It would be nice to get a sense of how much the proposed approach slows down training (e.g., report time per iteration with and without clustering). Finally, the description of how zero-shot clustering is performed is somewhat vague, and requires consulting the listed references. It would be nice to have a clearer description in the text.