More accurate machine learning models often demand more computation and memory at test time, making them difficult to deploy on CPU-or memoryconstrained devices. Model compression (also known as distillation) alleviates this burden by training a less expensive student model to mimic the expensive teacher model while maintaining most of the original accuracy. However, when fresh data is unavailable for the compression task, the teacher's training data is typically reused, leading to suboptimal compression. In this work, we propose to augment the compression dataset with synthetic data from a generative adversarial network (GAN) designed to approximate the training data distribution. Our GANassisted model compression (GAN-MC) significantly improves student accuracy for expensive models such as deep neural networks and large random forests on both image and tabular datasets. Building on these results, we propose a comprehensive metric-the Compression Score-to evaluate the quality of synthetic datasets based on their induced model compression performance. The Compression Score captures both data diversity and discriminability, and we illustrate its benefits over the popular Inception Score in the context of image classification.
Summary: 
The paper proposes an approach for improving standard techniques for model compression, i.e. compressing a big model (teacher) in a smaller and more computationally efficient one (student), using data generated by a conditional GAN (cGAN). The paper suggests that the standard practice of training the student to imitate the behavior of the teacher *on the same training data* that the teacher was trained on is problematic and can lead to overfitting. Instead, the paper proposes learning a conditional GAN, which can potentially generate large amounts of realistic synthetic data, and use this data (in addition to original training data) for model compression.
Experimental results show that this idea seems to improve the performance of convnet student models on CIFAR-10 classification and random forest student models on tabular data from UCI and Kaggle.
Another contribution of the paper is to propose an evaluation metric for generative model, called the compression score. This score evaluates the quality of generated data by using it in model compression: “good” synthetic data results in a smaller gap in performance between student and teacher models.
Strengths:
-	The paper sheds a light on an interesting aspect in model compression. The idea of teaching a student model to imitate behavior of the teacher model on *new* data is interesting. In fact, it emphasizes the fact that we are mostly interested in imitating the teacher model’s capability of generalizing to new examples rather than overfitting to training examples.
-	Experiments show that for several settings (model class, architecture and datasets), using synthetic data by a cGAN can be useful in reducing the gap between student and teacher models. 
-	The paper is clearly written and easy to follow.
Weaknesses:
-	The claim that reusing the same training data used for training the teacher model in model compression can lead to overfitting of student model is not very obvious and needs more experimental evidence in my opinion. One way to test this is to use some unseen real data (e.g. validation or a held-out part of training data) for model compression, and showing that it can indeed help in improving student performance.
-	The claim that cGAN can generate “infinite” amount of realistic data is too strong. In light of some well-known problems of GANs such as mode collapse [2] and low-support learned distributions [1], this assumption seems unrealistic. In fact, it is not too obvious how synthetic data by a generative model learned on *same training data as the teacher* can provide any additional information to real data.
-	While the idea of the proposed evaluation metric seems interesting, I believe it is not very practical, because: 
1.	It is computationally intensive (requires training a model from scratch on fake data)
2.	It relies on performance of the compression mechanism, which might also have some idiosyncrasies that prefer some features in synthetic data which do not necessarily correspond to quality of generated data.
Questions/Suggestions:
-	In addition to using held-out real data for model compression as suggested above, a useful baseline could be using standard data-augmentation techniques in model compression.
-	What would happen if a student model is very small and cannot possibly overfit training data? Would using synthetic data be still useful there?
-	I am actually confused about a claim made when presenting compression score in Section 5. The paper claims that the best compression score is 1 (training student model on real data), while the paper shows that in fact, good synthetic data should produce *better* accuracy than using real data. I would appreciate if authors can clarify this point.
Overall recommendation: 
While the paper presents an interesting problem in model compression, I’m leaning towards rejecting the paper because of the weaknesses mentioned above. That being said, I am happy to reconsider my decision if there is any misunderstanding on my part.
References:
[1] Arora, Sanjeev, and Yi Zhang. "Do GANs actually learn the distribution? an empirical study." arXiv preprint arXiv:1706.08224 (2017).
[2] Goodfellow, Ian. "NIPS 2016 tutorial: Generative adversarial networks." arXiv preprint arXiv:1701.00160 (2016).
-----
Updated score and posted a comment to author response.