Under what circumstances do local codes emerge in feed-forward neural networks
Localist coding schemes are more easily interpretable than the distributed schemes but generally believed to be biologically implausible. Recent results have found highly selective units and object detectors in NNs that are indicative of local codes (LCs). Here we undertake a constructionist study on feed-forward NNs and find LCs emerging in response to invariant features, and this finding is robust until the invariant feature is perturbed by 40%. Decreasing the number of input data, increasing the relative weight of the invariant features and large values of dropout all increase the number of LCs. Longer training times increase the number of LCs and the turning point of the LC-epoch curve correlates well with the point at which NNs reach 90-100% on both test and training accuracy. Pseudo-deep networks (2 hidden layers) which have many LCs lose them when common aspects of deep-NN research are applied (large training data, ReLU activations, early stopping on training accuracy and softmax), suggesting that LCs may not be found in deep-NNs. Switching to more biologically feasible constraints (sigmoidal activation functions, longer training times, dropout, activation noise) increases the number of LCs. If LCs are not found in the feed-forward classification layers of modern deep-CNNs these data suggest this could either be caused by a lack of (moderately) invariant features being passed to the fully connected layers or due to the choice of training conditions and architecture. Should the interpretability and resilience to noise of LCs be required, this work suggests how to tune a NN so they emerge.
The authors studied the local codes in neural networks through a set of controlled experiments (by controlling the invariance in the input data, dimensions of the input and the hidden layers, etc.), and identified some common conditions under which local codes are more likely to emerge. The fact that local codes tend to emerge as a response to invariance is interesting but not surprising, especially given that convolution operations are designed to capture location invariance. It would be useful if the authors can clarify their contributions and compare against existing works in the literature. Experiments are conducted at a relatively small scale: On a synthetic dataset with binarized vectors and on MNIST, which a predefined rule for noise injection (Figure 1). The controlled experiments conducted in the paper are still informative, but the overall message would be much stronger if the empirical analysis can be extended to common benchmarks such as CIFAR and/or ImageNet. All of the experiments are based on very shallow networks (3-4 layers), and as the result, the study ignores batch normalization and skip connections which are common ingredients in state-of-the-art convolutional networks. It remains unclear whether the presence of those components would change the emergence behavior of local codes, and hence affect some of the conclusions in the paper.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Under what circumstances do local codes emerge in feed-forward neural networks
Localist coding schemes are more easily interpretable than the distributed schemes but generally believed to be biologically implausible. Recent results have found highly selective units and object detectors in NNs that are indicative of local codes (LCs). Here we undertake a constructionist study on feed-forward NNs and find LCs emerging in response to invariant features, and this finding is robust until the invariant feature is perturbed by 40%. Decreasing the number of input data, increasing the relative weight of the invariant features and large values of dropout all increase the number of LCs. Longer training times increase the number of LCs and the turning point of the LC-epoch curve correlates well with the point at which NNs reach 90-100% on both test and training accuracy. Pseudo-deep networks (2 hidden layers) which have many LCs lose them when common aspects of deep-NN research are applied (large training data, ReLU activations, early stopping on training accuracy and softmax), suggesting that LCs may not be found in deep-NNs. Switching to more biologically feasible constraints (sigmoidal activation functions, longer training times, dropout, activation noise) increases the number of LCs. If LCs are not found in the feed-forward classification layers of modern deep-CNNs these data suggest this could either be caused by a lack of (moderately) invariant features being passed to the fully connected layers or due to the choice of training conditions and architecture. Should the interpretability and resilience to noise of LCs be required, this work suggests how to tune a NN so they emerge.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



The authors studied the local codes in neural networks through a set of controlled experiments (by controlling the invariance in the input data, dimensions of the input and the hidden layers, etc.), and identified some common conditions under which local codes are more likely to emerge. The fact that local codes tend to emerge as a response to invariance is interesting but not surprising, especially given that convolution operations are designed to capture location invariance. It would be useful if the authors can clarify their contributions and compare against existing works in the literature. Experiments are conducted at a relatively small scale: On a synthetic dataset with binarized vectors and on MNIST, which a predefined rule for noise injection (Figure 1). The controlled experiments conducted in the paper are still informative, but the overall message would be much stronger if the empirical analysis can be extended to common benchmarks such as CIFAR and/or ImageNet. All of the experiments are based on very shallow networks (3-4 layers), and as the result, the study ignores batch normalization and skip connections which are common ingredients in state-of-the-art convolutional networks. It remains unclear whether the presence of those components would change the emergence behavior of local codes, and hence affect some of the conclusions in the paper.
assistant
The authors studied the local codes in neural networks through a set of controlled experiments (by controlling the invariance in the input data, dimensions of the input and the hidden layers, etc. ), and identified some common conditions under which local codes are more likely to emerge. The fact that local codes tend to emerge as a response to invariance is interesting but not surprising, especially given that convolution operations are designed to capture location invariance. It would be useful if the authors can clarify their contributions and compare against existing works in the literature. Experiments are conducted at a relatively small scale: On a synthetic dataset with binarized vectors and on MNIST, which a predefined rule for noise injection (Figure 1). The controlled experiments conducted in the paper are still informative, but the overall message would be much stronger if the empirical analysis can be extended to common benchmarks such as CIFAR and/or ImageNet. All of the experiments are based on very shallow networks (3-4 layers), and as the result, the study ignores batch normalization and skip connections which are common ingredients in state-of-the-art convolutional networks. It remains unclear whether the presence of those components would change the emergence behavior of local codes, and hence affect some of the conclusions in the paper.