An adaptive homeostatic algorithm for the unsupervised learning of visual features
The formation of structure in the brain, that is, of the connections between cells within neural populations, is by large an unsupervised learning process: the emergence of this architecture is mostly self-organized. In the primary visual cortex of mammals, for example, one may observe during development the formation of cells selective to localized, oriented features. This leads to the development of a rough representation of contours of the retinal image in area V1. We modeled these mechanisms using sparse Hebbian learning algorithms. These algorithms alternate a coding step to encode the information with a learning step to find the proper encoder. A major difficulty faced by these algorithms is to deduce a good representation while knowing immature encoders, and to learn good encoders with a non-optimal representation. To address this problem, we propose to introduce a new regulation process between learning and coding, called homeostasis. Our homeostasis is compatible with a neuro-mimetic architecture and allows for the fast emergence of localized filters sensitive to orientation. The key to this algorithm lies in a simple adaptation mechanism based on non-linear functions that reconciles the antagonistic processes that occur at the coding and learning time scales. We tested this unsupervised algorithm with this homeostasis rule for a range of existing unsupervised learning algorithms coupled with different neural coding algorithms. In addition, we propose a simplification of this optimal homeostasis rule by implementing a simple heuristic on the probability of activation of neurons. Compared to the optimal homeostasis rule, we show that this heuristic allows to implement a more rapid unsupervised learning algorithm while keeping a large part of its effectiveness. These results demonstrate the potential application of such a strategy in machine learning and we illustrate this with one result in a convolutional neural network.
 This paper discusses the addition of a regularizer to a standard sparse coding/dictionary learning algorithm to encourage the atoms to be used with uniform frequency. I do not think this work should be accepted to the conference for the following reasons: 1: The authors show no benefit of this scheme except perhaps faster convergence. If faster training of dictionary learning models was a bottleneck in practical applications, this might be of interest, but it is not. SPAMS (http://spams-devel.gforge.inria.fr/) can train a model on image patches as the authors do here in a few tens of seconds on a modern computer. On the other hand, the authors give no evidence, empirical or otherwise, that their method is useful on any downstream tasks. In my view, they do not even show that the distribution of atom usage will be better with their algorithm after the learning has converged, as at least according to their learning curves, the baselines have not finished converging. It is not even clear that the final compression of the baselines would not be better. Even if they did show these convincingly, it is not obvious to me that it is valuable; the authors need to *show* that uniform usage is desirable. 2: The authors should compare against several costs/algorithms (e.g. l_0 with OMP, l_1 with LARS, etc.), and across various N_0/sparsity penalties, and across several datasets. The empirical evaluation is quite weak- one sparsity setting, two baselines, one dataset. Even without the "train to convergence" question above, I don*t think the authors have demonstrated that their claims on the properties of their algorithms/formulations are generally true.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
An adaptive homeostatic algorithm for the unsupervised learning of visual features
The formation of structure in the brain, that is, of the connections between cells within neural populations, is by large an unsupervised learning process: the emergence of this architecture is mostly self-organized. In the primary visual cortex of mammals, for example, one may observe during development the formation of cells selective to localized, oriented features. This leads to the development of a rough representation of contours of the retinal image in area V1. We modeled these mechanisms using sparse Hebbian learning algorithms. These algorithms alternate a coding step to encode the information with a learning step to find the proper encoder. A major difficulty faced by these algorithms is to deduce a good representation while knowing immature encoders, and to learn good encoders with a non-optimal representation. To address this problem, we propose to introduce a new regulation process between learning and coding, called homeostasis. Our homeostasis is compatible with a neuro-mimetic architecture and allows for the fast emergence of localized filters sensitive to orientation. The key to this algorithm lies in a simple adaptation mechanism based on non-linear functions that reconciles the antagonistic processes that occur at the coding and learning time scales. We tested this unsupervised algorithm with this homeostasis rule for a range of existing unsupervised learning algorithms coupled with different neural coding algorithms. In addition, we propose a simplification of this optimal homeostasis rule by implementing a simple heuristic on the probability of activation of neurons. Compared to the optimal homeostasis rule, we show that this heuristic allows to implement a more rapid unsupervised learning algorithm while keeping a large part of its effectiveness. These results demonstrate the potential application of such a strategy in machine learning and we illustrate this with one result in a convolutional neural network.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 This paper discusses the addition of a regularizer to a standard sparse coding/dictionary learning algorithm to encourage the atoms to be used with uniform frequency. I do not think this work should be accepted to the conference for the following reasons: 1: The authors show no benefit of this scheme except perhaps faster convergence. If faster training of dictionary learning models was a bottleneck in practical applications, this might be of interest, but it is not. SPAMS (http://spams-devel.gforge.inria.fr/) can train a model on image patches as the authors do here in a few tens of seconds on a modern computer. On the other hand, the authors give no evidence, empirical or otherwise, that their method is useful on any downstream tasks. In my view, they do not even show that the distribution of atom usage will be better with their algorithm after the learning has converged, as at least according to their learning curves, the baselines have not finished converging. It is not even clear that the final compression of the baselines would not be better. Even if they did show these convincingly, it is not obvious to me that it is valuable; the authors need to *show* that uniform usage is desirable. 2: The authors should compare against several costs/algorithms (e.g. l_0 with OMP, l_1 with LARS, etc.), and across various N_0/sparsity penalties, and across several datasets. The empirical evaluation is quite weak- one sparsity setting, two baselines, one dataset. Even without the "train to convergence" question above, I don*t think the authors have demonstrated that their claims on the properties of their algorithms/formulations are generally true.
assistant
 This paper discusses the addition of a regularizer to a standard sparse coding/dictionary learning algorithm to encourage the atoms to be used with uniform frequency. I do not think this work should be accepted to the conference for the following reasons: 1: The authors show no benefit of this scheme except perhaps faster convergence. If faster training of dictionary learning models was a bottleneck in practical applications, this might be of interest, but it is not. SPAMS (http://spams-devel.gforge.inria.fr/) can train a model on image patches as the authors do here in a few tens of seconds on a modern computer. On the other hand, the authors give no evidence, empirical or otherwise, that their method is useful on any downstream tasks. In my view, they do not even show that the distribution of atom usage will be better with their algorithm after the learning has converged, as at least according to their learning curves, the baselines have not finished converging. It is not even clear that the final compression of the baselines would not be better. Even if they did show these convincingly, it is not obvious to me that it is valuable; the authors need to *show* that uniform usage is desirable. 2: The authors should compare against several costs/algorithms (e.g. l_0 with OMP, l_1 with LARS, etc. ), and across various N_0/sparsity penalties, and across several datasets. The empirical evaluation is quite weak- one sparsity setting, two baselines, one dataset. Even without the "train to convergence" question above, I don*t think the authors have demonstrated that their claims on the properties of their algorithms/formulations are generally true.