A Normative Theory of Adaptive Dimensionality Reduction in Neural Networks
To make sense of the world our brains must analyze high-dimensional datasets streamed by our sensory organs. Because such analysis begins with dimensionality reduction, modeling early sensory processing requires biologically plausible online dimensionality reduction algorithms. Recently, we derived such an algorithm, termed similarity matching, from a Multidimensional Scaling (MDS) objective function. However, in the existing algorithm, the number of output dimensions is set a priori by the number of output neurons and cannot be changed. Because the number of informative dimensions in sensory inputs is variable there is a need for adaptive dimensionality reduction. Here, we derive biologically plausible dimensionality reduction algorithms which adapt the number of output dimensions to the eigenspectrum of the input covariance matrix. We formulate three objective functions which, in the offline setting, are optimized by the projections of the input dataset onto its principal subspace scaled by the eigenvalues of the output covariance matrix. In turn, the output eigenvalues are computed as i) soft-thresholded, ii) hard-thresholded, iii) equalized thresholded eigenvalues of the input covariance matrix. In the online setting, we derive the three corresponding adaptive algorithms and map them onto the dynamics of neuronal activity in networks with biologically plausible local learning rules. Remarkably, in the last two networks, neurons are divided into two classes which we identify with principal neurons and interneurons in biological circuits.
### Summary A normative approach is presented to derive local learning rules for a rate network that optimise a classic multidimensional scaling objective (MDS). The work extends a recent publication on the same topic in three ways: (1) it incorporates a low-rank regularisation on the network output, (2) adds another set of latent variables to match the distribution of Hebbian/Anti-Hebbian plasticity rules in cortical networks and (3) adds a penalty term to whiten the outputs.### Advances The manuscript represents a welcome addition to the (recently) growing set of publications that strive to derive local learning rules in a normative way. The ansatz presented here is interesting, though much of the novelty is already presented in the earlier publication [22]. The additions are nonetheless big enough to justify another publication. For one, it had so far been unclear whether low-rank regularisations can actually be incorporated into local learning rules, and the work here sets a nice precedence. Second, anti-hebbian rules are usually observed on inhibitory synapses, and so the extension by another set of latent variables with similar characteristics is welcome. ### Major First and foremost, the paper is not the first one to derive biologically plausible learning rule from a normative ansatz (as suggested in line 065). Besides the earlier publication on which this manuscript is build upon, there is at least the publication by Vertechi et al. (2014) [1] which derive neural networks and local learning rules for a rate-based neural networks that converges to an efficient population code (for both the over- as well as under complete case, the latter of which performs dimensionality reduction).The objectives (3) and (5) are difficult to understand in the way written here, and should be rather formulated in terms of the standard MDS with an additional nuclear norm regularisation on Y^TY. E. g. (3) is equivalent tomin_Y ||X^TX - Y^TY||_2^2 + alpha*T*||Y^TY||_*where ||.||_* is the nuclear norm (i.e. the sum of singular values). The authors note this relation in passing, but it would really help the reader if the intuition behind (3) and (5) is explained and formulated more extensively.More severely, I could not develop any intuition behind objective (7). This equation is essentially (5) without the quadratic terms, but there are no explanations whatsoever as to where this objective is coming from, what it is supposed to capture, and what it adds to the objectives (3) and (5).### Minor 1) Line 039 "each individual neuron still has to compress hundreds or thousands of its inputs into a single output". I do not doubt the importance of compression for the brain, but that is a very odd argument given that it is true for any linear transformation (being it expansive or contractive). 2) Line 044 "Because the signal-to-noise ratio of natural stimuli varies significantly [...]". I have no idea what that should mean - are you referring to measurement noise like in the retina? 3) Practical problems: how do you initialise the network?### Verdict The manuscript provides an interesting extension of a recent work on multidimensional scaling in plausible neural networks that can incorporate some rough statistics of synaptic plasticity as well as low-rank approximations using local learning rules alone. The manuscript could gain from more polishing, especially by providing intuition on the different objectives (3), (5) and (7).[1] Vertechi, Brendel & Machines, Unsupervised learning of an efficient short-term memory network, NIPS (2014)Update: I have seen the (helpful) response of the authors and I am confident that the points raised above will be sufficiently addressed during the revision.