Towards Understanding Learning Representations: To What Extent Do Different Neural Networks Learn the Same Representation
It is widely believed that learning good representations is one of the main reasons for the success of deep neural networks. Although highly intuitive, there is a lack of theory and systematic approach quantitatively characterizing what representations do deep neural networks learn. In this work, we move a tiny step towards a theory and better understanding of the representations. Specifically, we study a simpler problem: How similar are the representations learned by two networks with identical architecture but trained from different initializations. We develop a rigorous theory based on the neuron activation subspace match model. The theory gives a complete characterization of the structure of neuron activation subspace matches, where the core concepts are maximum match and simple match which describe the overall and the finest similarity between sets of neurons in two networks respectively. We also propose efficient algorithms to find the maximum match and simple matches. Finally, we conduct extensive experiments using our algorithms. Experimental results suggest that, surprisingly, representations learned by the same convolutional layers of networks trained from different initializations are not as similar as prevalently expected, at least in terms of subspace match.
Overall I liked the method developed by this paper, it’s an interesting alternative to the SVCCA method of [1]. The paper is very well written, and displays a high level of mathematical rigor. My main disappointment is the authors do not do much with their new technique, the main experimental observation is the middle layers of two independently trained VGG networks are not similar compared to the bottom and top layers. This is intriguing and I’m really curious as to why, do the authors have any insight into this? I’d also be curious to interpret what very small simple matches represent. One could for example pick the max variance linear combination of neurons in a simple match and visualize the validation images by projection on this subspace, if the concept of a simple match were meaningful then we might be able to interpret this by visualizing the images sorted along these projections.   I’d also like to see experiments regarding a sort of “null hypothesis” for this sort of analysis. For example, what happens if we run these similarity measures to compare a trained network to an untrained network, maybe we would still see matches just by spurious correlations. Did the authors investigate this? How large does the maximum match need to be before we should conclude the layers are similar?  Also how does this method account for the possibility of completely distributed representations? That is the method looks for subsets of neurons between two networks that have the same span. However, one could imagine a setting where there is a particular linear combination of all neurons in X that have the exact same span of a different linear combination of all neurons in Y, but no strict subset of X and Y match according to the definition in this work. In this case one it seems reasonable to conclude that these two linear combinations compute the same representation (at least assuming the match holds on enough data points), but the proposed method might not pick this up if no strict subset of X and Y have the same span. Can the authors comment on this possibility?   A few typos, and other comments:   31 “understand”  267 “neural networks”   272: The conclusion that the lower layers are not similar based on the fact that high layer similarity only occurs at epsilon=0.5 seems a bit heuristic to me. How do we measure similarity of layers as a function of epsilon and layer similarity? Can the authors give some insight as to how to interpret the size of epsilon and maximum matching similarity?   https://arxiv.org/abs/1706.05806  Edit: In light of the rebuttal I'm changing my score to 7. I did not see all of the additional experiments in the Supplementary material on my reading, and found the plots visualizing the simple matches between independently trained networks to be compelling. Good work!