The formation of structure in the brain, that is, of the connections between cells within neural populations, is by large an unsupervised learning process: the emergence of this architecture is mostly self-organized. In the primary visual cortex of mammals, for example, one may observe during development the formation of cells selective to localized, oriented features. This leads to the development of a rough representation of contours of the retinal image in area V1. We modeled these mechanisms using sparse Hebbian learning algorithms. These algorithms alternate a coding step to encode the information with a learning step to find the proper encoder. A major difficulty faced by these algorithms is to deduce a good representation while knowing immature encoders, and to learn good encoders with a non-optimal representation. To address this problem, we propose to introduce a new regulation process between learning and coding, called homeostasis. Our homeostasis is compatible with a neuro-mimetic architecture and allows for the fast emergence of localized filters sensitive to orientation. The key to this algorithm lies in a simple adaptation mechanism based on non-linear functions that reconciles the antagonistic processes that occur at the coding and learning time scales. We tested this unsupervised algorithm with this homeostasis rule for a range of existing unsupervised learning algorithms coupled with different neural coding algorithms. In addition, we propose a simplification of this optimal homeostasis rule by implementing a simple heuristic on the probability of activation of neurons. Compared to the optimal homeostasis rule, we show that this heuristic allows to implement a more rapid unsupervised learning algorithm while keeping a large part of its effectiveness. These results demonstrate the potential application of such a strategy in machine learning and we illustrate this with one result in a convolutional neural network.
Please consider this rubric when writing your review:
1. Briefly establish your personal expertise in the field of the paper.
2. Concisely summarize the contributions of the paper.
3. Evaluate the quality and composition of the work.
4. Place the work in context of prior work, and evaluate this work's novelty.
5. Provide critique of each theorem or experiment that is relevant to your judgment of the paper's novelty and quality.
6. Provide a summary judgment if the work is significant and of interest to the community.
1. I am a researcher working at the intersection of machine learning,
computational neuroscience and biological vision.  I have experience
with neural network models and visual neurophysiology.
2. This paper develops and tests an adaptive homeostatic algorithm for
unsupervised visual feature learning (for example for learning models
of early visual processing/V1).
3.The work spends a lot of pages describing the general problem of
unsupervised feature learning and the history of the base algorithms.
The literature review is quite extensive.  The new content appears to
be in section 2.2 (Histogram Equalization Homeostasis - HEH), where a
simple idea to keep all units with balanced activity over the set of
natural images.  The authors also develop a computationally cheaper
version they call HAP (Homeostasis on Activation Probability) The
authors show that their F function is optimized quicker with the HEH
and HAP algorithms.  I would like to see how these curves vary with
the number of neurons (e.g. can you add X% more neurons and get
similar convergence speed -- and if so which is more computationally
costly)?
4. Many groups have developed various homeostatic algorithms for
unsupervised learning, though I have not seen this exact one before.
5.  The experiments reveal the resulting receptive fields and show the 
decrease in the F function (error function).    The resulting receptive fields
do not seem that different to me between the different methods.  I am also not
that convinced that the faster convergence as a function of learning step is that important
especially as the learning steps may be more computationally expensive for this method.
6. I am not sure how interesting this work will be for the ICLR audience,
as it is not clear how important the faster convergence and more even
utilization of neurons is (and how it would compare computationally
with just having more neurons).