Role-Wise Data Augmentation for Knowledge Distillation
Knowledge Distillation (KD) is a common method for transferring the "knowledge" learned by one machine learning model (the teacher) into another model (the student), where typically, the teacher has a greater capacity (e.g., more parameters or higher bit-widths). To our knowledge, existing methods overlook the fact that although the student absorbs extra knowledge from the teacher, both models share the same input data -and this data is the only medium by which the teacher's knowledge can be demonstrated. Due to the difference in model capacities, the student may not benefit fully from the same data points on which the teacher is trained. On the other hand, a human teacher may demonstrate a piece of knowledge with individualized examples adapted to a particular student, for instance, in terms of her cultural background and interests. Inspired by this behavior, we design data augmentation agents with distinct roles to facilitate knowledge distillation. Our data augmentation agents generate distinct training data for the teacher and student, respectively. We focus specifically on KD when the teacher network has greater precision (bit-width) than the student network. We find empirically that specially tailored data points enable the teacher's knowledge to be demonstrated more effectively to the student. We compare our approach with existing KD methods on training popular neural architectures and demonstrate that role-wise data augmentation improves the effectiveness of KD over strong prior approaches. The code for reproducing our results will be made publicly available.
The authors hypothesise that, in a Knowledge Distillation (KD) setting, the student network may benefit from learning from different training data than the teacher. The motivation comes from the fact that human teachers "adapt" the examples given their students, depending on their individual expertise on the subject, personal and cultural biases, and other factors.
More concretely, the paper proposes to use an evolutionary algorithm for data augmentation (PBA, already published) to train the teacher and student networks. The key aspect of the proposed method is that the teacher and the student use different augmentation schedules. The augmentation schedules improve results on both the teacher and student networks.
Orthogonally to this, they also propose to combine different KD objective functions and show that this also improves the results over each loss in isolation. 
The datasets used in the experiments are CIFAR-10 and CIFAR-100. In most of the experiments the student has a (much) lower bit-width than the teacher, but they also apply the method with a student network with less parameters. Two different network architectures are used in the experiments: AlexNet and Resnet18, their method shows consistent improvements over the baselines in all cases (although some improvements may be considered marginal).
Overall the paper is well motivated, clearly written and, in the experiments section, they give enough details, and/or cite previous works that contain them, to reproduce the experiments.
Most importantly, the experiments are well designed to test the initial hypothesis: 
1. The authors show that the student (and teacher) trained with the PBA data augmentation achieves a higher accuracy than the baseline method (Table 2). However, this is not enough to confirm/refute the hypothesis itself, since it is known that data augmentation generally helps.
2. They show that the two augmentation strategies are different, by using the teacher’s augmentation to train the student network (Table 3). The results show that it’s better to use a specific data augmentation schedule for the student network.
However, (the main criticism is that) the paper only partially confirms the hypothesis. For instance, it is not clear whether the hypothesis is true for other forms of KD, or it only applies when training a student with lower bit-width. In Section 5.6, they train a student with fewer parameters (less layers) than the teacher, instead of fewer bits/parameter. However, the improvements of their method over the baseline (II-KD) seem marginal there. And most importantly, as pointed out earlier, this alone is not enough to confirm the hypothesis. A similar table to Table 3 should be included in this section.
Secondly, the proposed KD loss, which is just a combination of previously published works, is an orthogonal improvement to the main method (as the authors admit), and it is not related at all to the subject of the study. It is obviously good to introduce more than one contribution in a paper, but this second (and minor) contribution should be well motivated in its own, in order to avoid "distracting" the reader from the main contribution.
In addition, I would suggest to perform statistical tests to give additional robustness to the conclusions drawn from the experiments. They perform experiments on different architectures (AlexNet and Resnet18) and different KD losses, and the conclusions are always consistent with the hypothesis, but it’s not clear whether the improvements are statistically significant or not. In this regard, it would also be appreciated to include results with other datasets, since only CIFAR-10 and CIFAR-100 (which are very similar datasets) were used. 
Beyond statistical significance, it’s also not clear how important are these results for researchers working outside the scope of Knowledge Distillation. 
Finally, some minor comments:
- L_original in Eq. (9) is not defined. I'm assuming it's the L_{KD}^{soft} loss.
- Please, check consistency of pronouns: authors refer to the teacher network as “it” (e.g. “[...] teacher distills knowledge to a narrow/shallow student to improve its performance”, page 2), but to the student as “she” (e.g. “[...] to train the student better from her teacher”, page 2).
- “our methods clearly outperforms [...]” (page 6)  -> “our method clearly outperforms [...]”.
Score: Borderline accept, but I will increase the score if the authors address my concerns and provide better evidence that the hypothesis is confirmed.