Attentional Neural Network: Feature Selection Using Cognitive Feedback
Attentional Neural Network is a new framework that integrates top-down cognitive bias and bottom-up feature extraction in one coherent architecture. The top-down influence is especially effective when dealing with high noise or difficult segmentation problems. Our system is modular and extensible. It is also easy to train and cheap to run, and yet can accommodate complex behaviors. We obtain classification accuracy better than or competitive with state of art results on the MNIST variation dataset, and successfully disentangle overlaid digits with high success rates. We view such a general purpose framework as an essential foundation for a larger system emulating the cognitive abilities of the whole brain.
The authors introduce the Attentional Neural Network which integrates top-downbias and bottom-up feature extraction in a single architecture.I like very much the idea of integrating top-down connections, but I have someconcerns on how these have been implemented in the proposed approach and onthe evaluation protocol.Generally the paper reads well. I strongly recommend to run a spell-check (Ifound many typos) and to rewrite few convoluted sentences. Please always favorlinear and short sentences.The denoising idea to segment out some patterns of interest to be fed to aclassifier is interesting.It needs to be said that having this gated interaction is not completely noveland there are already studies on gated RBM and gated Autoencoder. However, the cognitive bias is a binary N (number of classes) dimensionalvector, and this requires to perform exhaustive search.Why isn't the bias function of some intermediate representation? Also, having a separate training for each of the modules makes tricky to assessthe best strategy to "confirm the initial b guess", as also stated in theparagraph beginning at line 113. This is in my opinion a weak point of thework which makes it unfeasible for many problems.I always prefer end-to-end systems, at least here the classifier could havebeen trained jointly with the M module, even following the proposed alternatingminimization scheme.The classification experiment is fine, but it is not clear to me why theauthors did not train the classifier C on the same data. Abaseline is paramount to evaluate the relative improvement of the proposedapproach; how does aNN perform when compared with the simple classifier Cused after M?The MNIST-2 problem is interesting. The proposed way to disentangle thevarious digits is a bit tricky though because of the "validate the guess"criterion, which is not easy to be selected.If we are interested in classification, again, what is the performance of Cwhen taking the first two best guesses (naive) or when C is trained withcross-entropy to predict the presence of given objects?Even following the proposed scheme, what if the number of objects is notknown? How to set the criterion? Let's say we were to apply it to a generalobject detection benchmark.Overall I like the approach but there are important experiments which need tobe performed to correctly evaluate the relative improvement of aNN.Thanks for the work.