Manifold-tiling Localized Receptive Fields are Optimal in Similarity-preserving Neural Networks
Many neurons in the brain, such as place cells in the rodent hippocampus, have localized receptive fields, i.e., they respond to a small neighborhood of stimulus space. What is the functional significance of such representations and how can they arise? Here, we propose that localized receptive fields emerge in similarity-preserving networks of rectifying neurons that learn low-dimensional manifolds populated by sensory inputs. Numerical simulations of such networks on standard datasets yield manifold-tiling localized receptive fields. More generally, we show analytically that, for data lying on symmetric manifolds, optimal solutions of objectives, from which similarity-preserving networks are derived, have localized receptive fields. Therefore, nonnegative similarity-preserving mapping (NSM) implemented by neural networks can model representations of continuous manifolds in the brain.
In this work, the authors show that similarity-preserving, the problem of creating an encoding of inputs on a manifold such that the similarity between encoded outputs between inputs matches similarity between inputs, yields localized receptive field that tile the space. They present a biologically plausible online neural learning model which combines several neural elements in a way that is equivalent to optimizing performance on this problem, and thus learns localized receptive fields. Additionally, they apply these methods to generate localized receptive fields over real-world datasets, such as MNIST and images of rotated objects.   Overall I found this to be a good paper and enjoyed reading it.   Quality:  The paper starts from the basic, reasonable idea of similarity-preserving transformations, and then proceeds to prove mathematically, using sophisticated techniques, that the optimal receptive fields are localized, as well as show that these can be optimized by a biologically plausible learning rule.   I view this as a significant result, although I don’t know the literature so well.  The experimental results with MNIST and rotated objects are interesting, especially in that the manifold topology of inputs matches the manifold topology of encodings, but there is a somewhat preliminary feel to them, in that there is not much information about how well this scheme is able to generalize to unknown examples.   Clarity:  While I was not able to follow all proofs and mathematical detail, the authors use some fairly heavy-duty techniques, so I don’t think that’s a fair criticism. Given how dense the proof techniques are, I feel that the paper would be significantly improved if the authors took a more pedagogical/intuition-providing approach in general.  For example, in Line 130, I have to admit I did not understand the proof of theorem 1, although it seems quite technical. I have some intuition for why this is true, but it might be helpful for the authors to provide some intuition. For example, there might be a much simpler an intuitive proof for the ring in the continuum limit; would it be possible for the authors to demonstrate this special case in a more pedagogical manner? line 184 Section 5: After staring for a little while, I saw how NSM-1 yields equation 11, but still don’t understand how the authors get from equation (11) to equation (12). The authors cite:  Cengiz Pehlevan, Anirvan M Sengupta, and Dmitri B Chklovskii. Why do similarity matching objectives lead to hebbian/anti-hebbian networks? Neural computation, 30(1):84–124, 2018  nearby but there seems is no proof there either. Can the authors add a section in the supplemental proving that this is true and add more intuition behind what each of these terms are? Some intuition is provided in the main, but it is quite cursory given space limitations.  During online optimization(line 188), what values were used for relative learning rates gamma and eta? My understanding of equations (13) (14), is that each step relies on a *single* sample x_t. Is this correct? If so, is the manner in which the manifold is explored important? For example, does it depend if x_{t}, x_{t+1} tend to be similar or unrelated?  There seem to be other details missing, which is why I gave the current manuscript a medium reproducibility score rather than a high one. I suspect these details can be fixed and filled in without too much trouble.  Originality and Significance:  While I don’t claim to have a deep knowledge of the literature, this work seems to be original and significant. Receptive fields localized in physical/configuration space are found in many places, and understanding their function as well as their formation through the lens of similarity-preservation might be an important insight.   There seem to be analytical results that exist concerning related similarity-preserving optimization problems, but not this particular one involving a cutoff where nearly-orthogonal inputs should yield completely-orthogonal encodings.    Miscellaneous:  In figure 4, how is the receptive field width and density (Y transpose, third matrix) determined by the input correlation structure (X transpose X, first matrix)? Can the authors provide some intuition behind this? It would also be nice if the authors provided some intuition/supplementary figures for how receptive fields depend on various parameters in the manner of SI figure 6.