This report introduces a training and recognition scheme, in which classification is realized via class-wise discerning. Trained with datasets whose labels are randomly shuffled except for one class of interest, a neural network learns class-wise parameter values, and remolds itself from a feature sorter into feature filters, each of which discerns objects belonging to one of the classes only. Classification of an input can be inferred from the maximum response of the filters. A multiple check with multiple versions of filters can diminish fluctuation and yields better performance. This scheme of discerning, maximum response and multiple check is a method of general viability to improve performance of feedforward networks, and the filter training itself is a promising feature abstraction procedure. In contrast to the direct sorting, the scheme mimics the classification process mediated by a series of one component picking.
The paper discusses a method to increase accuracy of deep-nets on multi-class classification tasks by what seems to be a reduction of multi-class to binary classification following the classical one-vs-all mechanism. I fail to see any novelty in the paper. The problem of reducing multi-class to binary classification has been studied thoroughly with many classical papers like:
1. Usual one-vs-all - this paper does the same thing as one vs all in my opinion even though this technique is known for a decade or so. 
2. Weighted One-Against-All - http://hunch.net/~jl/projects/reductions/woa/woa.pdf
and more sophisticated techniques like:
3. Alina Beygelzimer, John Langford, Pradeep D. Ravikumar. Error-Correcting Tournaments. CoRR, abs/0902.3176, 2009.
4. Erin L. Allwein, Robert E. Schapire, Yoram Singer, Pack Kaelbling. Reducing Multiclass to Binary: A Unifying Approach for Margin Classifiers. Journal of Machine Learning Research, 113—141, 2000.
5. Thomas G. Dietterich, Ghulum Bakiri. Solving multiclass learning problems via error-correcting output codes. Journal of Artificial Intelligence Research, 2:263—286, 1995.
In my opinion the methods in [1,2,3,5] above can be used with any binary learners and therefore deep-networks. This paper makes no effort in comparing with any of these well-known papers. Moreover the experiments do not show any gain in state of the art performances in the data-sets used, as experiments are done with toy-networks. Further some rules for selecting the class is discussed in Section 3. There are many known rules for generating probability scores in one-vs-all classification and the relations to these are not discussed. 
Therefore, I fail to see any novelty in this paper (theoretical or empirical).