We introduce advocacy learning, a novel supervised training scheme for classification problems. This training scheme applies to a framework consisting of two connected networks: 1) the Advocates, composed of one subnetwork per class, which take the input and provide a convincing class-conditional argument in the form of an attention map, and 2) a Judge, which predicts the inputs class label based on these arguments. Each Advocate aims to convince the Judge that the input example belongs to their corresponding class. In contrast to a standard network, in which all subnetworks are trained to jointly cooperate, we train the Advocates to competitively argue for their class, even when the input belongs to a different class. We also explore a variant, honest advocacy learning, where the Advocates are only trained on data corresponding to their class. Applied to several different classification tasks, we show that advocacy learning can lead to small improvements in classification accuracy over an identical supervised baseline. Through a series of follow-up experiments, we analyze when and how Advocates improve discriminative performance. Though it may seem counter-intuitive, a framework in which subnetworks are trained to competitively provide evidence in support of their class shows promise, performing as well as or better than standard approaches. This provides a foundation for further exploration into the effect of competition and class-conditional representations.
This paper presents a novel concept of supervised learning, advocacy learning. In this framework, supervised learning procedure is given by two subnetworks, advocates and judge. Advocates generate evidence in the form of attention for individual classes and judge decide the final class labels.
The main idea looks interesting, and the paper is clear enough to deliver the idea. However, this paper has the following major issues.
1. There is no formal justification of the idea. Although the idea looks interesting, there is no theoretical background and no clear intuition.
2. Experiment is weak and even inconsistent. Evaluation is performed on very small datasets only, where all baseline methods already show very high accuracy and accuracy gain given by the proposed method is very marginal. In particular, Table 1 and 2  have inconsistent results; advocacy network is better in Table 1 while worse in Table 2 compared to honest advocacy network. To make the idea more convincing, it is required to test it on much larger datasets, at least ImageNet scale, and more desirable to show results in other tasks such as object detection and image segmentation.
3. I am not sure if advocacy network has any separate supervision to enforce it to be learned in a class-conditional manner. Also, in honest advocacy network, each subnetwork can look at only a part of dataset (data corresponding to the class), and I wonder if there is any problem given by data deficiency issue.
Overall, the paper does not look ready for publication because the idea is clearly justified neither theoretically nor empirically.