More accurate machine learning models often demand more computation and memory at test time, making them difficult to deploy on CPU-or memoryconstrained devices. Teacher-student compression (TSC), also known as distillation, alleviates this burden by training a less expensive student model to mimic the expensive teacher model while maintaining most of the original accuracy. However, when fresh data is unavailable for the compression task, the teacher's training data is typically reused, leading to suboptimal compression. In this work, we propose to augment the compression dataset with synthetic data from a generative adversarial network (GAN) designed to approximate the training data distribution. Our GAN-assisted TSC (GAN-TSC) significantly improves student accuracy for expensive models such as large random forests and deep neural networks on both tabular and image datasets. Building on these results, we propose a comprehensive metric-the TSC Score-to evaluate the quality of synthetic datasets based on their induced TSC performance. The TSC Score captures both data diversity and class affinity, and we illustrate its benefits over the popular Inception Score in the context of image classification.
This paper presents an algorithm to generate images for training a student network through distillation. The paper claims the use of the same training data is not necessarily beneficial when it comes to improving the accuracy of the student being trained. 
The paper sits on the empirically driven side with sufficient experiments in the context of random forest and CNN.
As a second contribution, the paper proposes a scoring process to evaluate the quality of datasets generated by GAN methods. There are little experiments on this side. 
Pros:
- I like the paper and the idea behind being able to improve or even train a student network when the original data is not present.
- Metrics tailored to the problem are relevant. 
Negs:
- While there are plenty of experiments, there is a lack of detailed descriptions. 
- The scoring for GANS seems to be barely tested. In the scoring GAN, The fact that TSCS drops is enough to be a valid metric (or better than existing?). In the TSC vs inception, it is hard for me to see the unrealistic artifacts and, according to the text, that is not what TSCS is measuring, right? What is the influence of using different student-teacher configuration? (on the time to produce the scores the paper claims NiN and LeNet, is there any difference if using other architectures (ResNet family for instance)? At least, in that case, the time changes. It would be nice to see the stability of this metric to demonstrate that the need for training an inexpensive model is correct. 
For the TSC vs Inception, the GANs are subjectively assessed, isn't it (as to select well-trained to Inferior). Would it be possible to see exactly the same images between the two of them? Seems like the difference in quality for IS is significantly larger than for the proposed metric (even in the last gan there is a slight increase in the metric). 
It seems to me that IS is just a quality metric based on how a single image looks like. Would it be possible to disentangle the training and the scoring in the proposed metric? What if my hyperparameters for doing the single epoch are totally wrong?
In the general idea of the GAN, while I like it, there is little about how the GAN is actually trained. I guess this GAN is trained using some sort of real data and therefore, the comparison is not totally fair. How many images were used to train this GAN? What would happen if those images are used directly in the distillation framework? 
How many images are generated in section 4? Is the influence of pfake related to the dataset? If I have to train using another dataset, how do i set that parameter?