Compete to Compute
Local competition among neighboring neurons is common in biological neural networks (NNs). In this paper, we apply the concept to gradient-based, backprop-trained artificial multilayer NNs. NNs with competing linear units tend to outperform those with non-competing nonlinear units, and avoid catastrophic forgetting when training sets change over time.
This paper introduces a simple form of nonlinearity 
      into neural net architectures - forming groups of (typically two) neurons 
      and zeroing out all the neurons in a group except the one with the highest 
      value. While a very simple idea, it seems to give good results on 
      classification. The authors also give evidence that the network does not 
      forget as much as networks with more standard nonlnearities, however there 
      might be a problem with that experiment (see below). This paper 
      introduces a simple form of nonlinearity into neural net architectures - 
      forming groups of (typically two) neurons and zeroing out all the neurons 
      in a group except the one with the highest value. While a very simple 
      idea, it seems to give good results on classification. The authors also 
      give evidence that the network does not forget as much as networks with 
      more standard nonlnearities, however there might be a problem with that 
      experiment (see below). 1) There are better results on permutation 
      invariant mnist, see table 1. in http://arxiv.org/pdf/1302.4389v3.pdf 
      .Some of them are just feedforward networks. Also I don't agree that 
      droput in the input should be considered a data augmentation since it 
      doesn't assume anything about the input structure. You should have tried 
      that experiment too. 2) It is a good property for the network not 
      to forget. However the experiments could have few issues. You wait until 
      network reaches certain likelihood and then change the data/labels. Since 
      the new nonlinearity peforms better on recognition, it doesn't have to 
      work as hard to reach the likelihood and so it doesn't need to do so much 
      training, and so the reason it doesn't forget as much can simply be that 
      it didn't train as much to forget what it has learned. 3) You 
      should have also done a similar and related experiment. You have obtained 
      the perofmance when you train on all digits at the same time (till 
      convergence). Now, train till convergence one digits 1,2,3,4,5, then add 
      the remaining digits and train on all digits. This tests how much is the 
      network stuck in the minimum it found when training in the first phase. 
      Ideally the performance in the second experiment (training on 1-5 and then 
      training on 1-10) is the same as training on 1-10 from the start. It is 
      known that this is not the case for sigmoid networks. Quality: 
      Good, but more experiments should be there and the forgeting expriment 
      should be better. Clarity: Very good Originality: I haven't seen 
      it before. On negative side it is just another nonlinearity, related to 
      max pooling, on the positive side it is a simple idea the gives good 
      result. Significance: Somewhat significant - another nonlinearity into 
      neural networks toolbox.