Associative Memory via a Sparse Recovery Model
An associative memory is a structure learned from a dataset M of vectors (signals) in a way such that, given a noisy version of one of the vectors as input, the nearest valid vector from M (nearest neighbor) is provided as output, preferably via a fast iterative algorithm. Traditionally, binary (or q-ary) Hopfield neural networks are used to model the above structure. In this paper, for the first time, we propose a model of associative memory based on sparse recovery of signals. Our basic premise is simple. For a dataset, we learn a set of linear constraints that every vector in the dataset must satisfy. Provided these linear constraints possess some special properties, it is possible to cast the task of finding nearest neighbor as a sparse recovery problem. Assuming generic random models for the dataset, we show that it is possible to store super-polynomial or exponential number of n-length vectors in a neural network of size O(n). Furthermore, given a noisy version of one of the stored vectors corrupted in near-linear number of coordinates, the vector can be correctly recalled using a neurally feasible algorithm.
Associative memory via a Sparse Recovery Model -------------------------------------------------- This paper builds an associative memory using message passing in a random graph setting.For an appropriately chosen set of constraints one can create an associative memory that seems to perform well using only local computations.I think the task is valid and the solution is interesting.I would have preferred more experimental work, so I am recommending a weak accept.More technical comments: ----------------------- Minor quibble - on line 46 - please refer to 1.22^n as exp(0.2*n).It's just more consistent with the rest of your notation in the rest of the paperOn line 95, you mention that your model is robust against adversarial error patterns but then don't mention it again.Can you please go into more detail about this claim or drop it entirely?Regarding equation 20, you casually mention that it is a neurally feasible algorithm since it involves only matrix vector multiplications.However matrix B appears twice in the equation, which seems to imply that there is a non-local dependency in your algorithm. (i.e. if one were to change a tweak a single parameter in the B matrix then one would have to modify your neural implementation in two, non local places).This makes it hard to see how such an algorithm could be learnt.If you do have any ideas n how this could be learnt then it would be great to add them, as it would really strengthen the paper.For your experimental section I would like to see more evidence showing that scaling behaviour that you predict does indeed show up in practice as you scale the network out.(i.e. iterative soft thresholding doesn't change the theoretical analysis.)It would also be great to see information theoretic performance measures.e.g. how does the mutual information between the initial cue and true pattern compare with the mutual information between the recalled pattern and the true pattern?Please consider also citing: Sterne (2012) - Efficient and Robust Associative Memory from a Generalized Bloom Filter which also does associative memory using a random graph with message passing.If you think the paper is very different then I won't push the point, but it seemed relevant to me.