Feature maps contain rich information about image intensity and spatial correlation. However, previous online knowledge distillation methods only utilize the class probabilities. Thus in this paper, we propose an online knowledge distillation method that transfers not only the knowledge of the class probabilities but also that of the feature map using the adversarial training framework. We train multiple networks simultaneously by employing discriminators to distinguish the feature map distributions of different networks. Each network has its corresponding discriminator which discriminates the feature map from its own as fake while classifying that of the other network as real. By training a network to fool the corresponding discriminator, it can learn the other network's feature map distribution. Discriminators and networks are trained concurrently in a minimax twoplayer game. Also, we propose a novel cyclic learning scheme for training more than two networks together. We have applied our method to various network architectures on the classification task and discovered a significant improvement of performance especially in the case of training a pair of a small network and a large one.
= Summary
This paper presents a new deep mutual learning (i.e., online peer-teaching) method based on Knowledge Distillation (KD) in a feature map level. The target task is similar with the original KD in the sense that the a network is taught by another network as well as groundtruth labels, but different with the KD in the sense that the networks are not a (frozen) teacher and a student but teaching each other in an online manner. Most approaches in this relatively new line of research rely on logit-based KD for transferring knowledges between networks, and the paper demonstrates that by an additional feature map level KD the performance can be further improved.
= Decision
The current decision is borderline in my mind, but officially weak accept. Although the proposed method is simple and consists of known ideas, it is designed convincingly and enhances performance practically. Also, I believe the target task itself is worth to be introduced as a next direction of KD. However, the submission is weak in terms of novelty and the manuscript should be polished carefully.
= Comments
1) Weak clarity
- The main motivation and advantages of the deep mutual learning is not well introduced in Section 1. Although the two papers (i.e., ONE and DML) are cited here, it would be much better to explicitly describe the main idea and motivation of peer-teaching, what is the difference between the task and the original KD, and the achievements in the previous work. Without these, readers, including me, may get confused why the online KD is required and why there is no clear teacher-student relationship between networks. 
- In a similar context, the motivation of introducing more than two student networks should be given.
- The architecture of the discriminator seems not described even in the appendix. 
- The meaning of various arrow types in Figure 1 is not clearly described.
2) Insufficient experiments
It would be better to report the performance of vanilla and (offline) KD in Table 1 to show more clearly that the feature map alignment is useful and that online KD is better than its offline counterpart.
3) Limited novelty and performance improvement
- The main idea is already introduced in previous work on the task and the feature map level KD has been studied widely for various applications, their combination is somewhat new though.
- The performance improvement by the proposed feature map level KD seems marginal as shown in Table 2.
- The performance gap between DML and the proposed model seems also marginal.