Dataset Distillation
Model distillation aims to distill the knowledge of a complex model into a simpler one. In this paper, we consider an alternative formulation called dataset distillation: we keep the model fixed and instead attempt to distill the knowledge from a large training dataset into a small one. The idea is to synthesize a small number of data points that do not need to come from the correct data distribution, but will, when given to the learning algorithm as training data, approximate the model trained on the original data. For example, we show that it is possible to compress 60,000 MNIST training images into just 10 synthetic distilled images (one per class) and achieve close to the original performance, given a fixed network initialization. We evaluate our method in various initialization settings. Experiments on multiple datasets, MNIST, CIFAR10, PASCAL-VOC, and CUB-200, demonstrate the ad-vantage of our approach compared to alternative methods. Finally, we include a real-world application of dataset distillation to the continual learning setting: we show that storing distilled images as episodic memory of previous tasks can alleviate forgetting more effectively than real images.
This paper proposes to distill a dataset into a small set of artifical images, which can be used to then train a neural network. The proposed method of doing this is based on a meta-learning flavored formulation where a GD step is done on the synthetic images and then the loss computed on the real images. The procedure appears to overfit to the initialization of the classification model and thus to resolve this the authors randomly resample the initialization throughout training. Experimental results compare the method to other coreset construction strategies and random sampling. Applications are also shown to continual learning. I think the general idea of this paper and the questions it poses are very interesting and in fact a rather underexplored area however the experimental results are quite underwhelming. The original questions posed in the paper are not really answered (can we distill a dataset?). The experiments only demonstrate that an expensive procedure that trains a model can produce synthetic images that give higher performance than a randomly selected set of images. Comments regarding experiments from Sec 4.1 The results indeed show the performance is greatly improved over using random images from the dataset, however obtaining 45% on CIFAR-10 would hardly be considered “distilling the dataset” as the full set of data yields performances of 90%+ on this task. I would be more interested to see how many samples are needed to distill the dataset such that it reaches reasonably performance. It is not completely clear why the authors didn’t do this kind of analysis but it might be due to the heavy computational cost of this method, in which case it would be good to discuss that. Similarly for MNIST the performance is quite low (effective use of regularization and can yield similar performance with a 100 images), kmeans++ based selection gives quite close results to the synthetic procedure for MNIST. I think its also a bit misleading that the authors advertise the fixed initialization performance in the abstract, this performance in my mind is mainly useful for the purposes of analyzing the method. I would also be interested to know if the coreset construction baselines can be improved. For example the kmeans++, why couldnt it be done in the feature space of a trained model (since synthetic method also trains a model). I am not an expert in the area of coresets but it would be good to know what is the state-of-the-art. A potentially interesting coreset consturction approach that also requires training a classifier can be found in this ICLR 2019 paper https://arxiv.org/pdf/1812.05159.pdf Comments regarding experiments for the continual learning application: I posted a separate comment regarding this set of experiments, which I hope the authors can address. Currently it is not 100% clear what is done. For now I will go under the assumptions that the authors are ingesting the entire dataset for each new task, using this to get their distilled data. This seems like an unfair baseline to me for the GEM setup which sees each data point only once and a for a single iteration. The authors introduce an additional step which requires multiple iterations over the whole dataset to obtain the synthetic set of images. This may be more appropriate for comparison in another continual learning setting. A few other observations/suggestions - In my understanding the label of the synthetic data is currently fixed. It would be interesting if this can also be adapted or alternatively if a fixed soft label (e.g 0.3 one class 0.7) can be assigned. - I would be interested to know if the target architecuture were different would the results change? --- Addendum: After writing this review the authors have responded to my concerns regarding the CL experiments and have begun to recompute some of the experiments using a more rigorous comparison with promising results on permuted mnist. If the CL section is clarified in the revised version (I suggest besides more decriptions in the main text to perhaps add an explicit algorithm block in the appendix) and the more rigorous comparison (single epoch to signl epoch) extend to cifar100 I will consider increasing my score as the method will be competitive for CL. I do still think the non-CL experiments (which make up the bulk of the paper are somewhat underwhelming as discussed in the review)
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Dataset Distillation
Model distillation aims to distill the knowledge of a complex model into a simpler one. In this paper, we consider an alternative formulation called dataset distillation: we keep the model fixed and instead attempt to distill the knowledge from a large training dataset into a small one. The idea is to synthesize a small number of data points that do not need to come from the correct data distribution, but will, when given to the learning algorithm as training data, approximate the model trained on the original data. For example, we show that it is possible to compress 60,000 MNIST training images into just 10 synthetic distilled images (one per class) and achieve close to the original performance, given a fixed network initialization. We evaluate our method in various initialization settings. Experiments on multiple datasets, MNIST, CIFAR10, PASCAL-VOC, and CUB-200, demonstrate the ad-vantage of our approach compared to alternative methods. Finally, we include a real-world application of dataset distillation to the continual learning setting: we show that storing distilled images as episodic memory of previous tasks can alleviate forgetting more effectively than real images.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



This paper proposes to distill a dataset into a small set of artifical images, which can be used to then train a neural network. The proposed method of doing this is based on a meta-learning flavored formulation where a GD step is done on the synthetic images and then the loss computed on the real images. The procedure appears to overfit to the initialization of the classification model and thus to resolve this the authors randomly resample the initialization throughout training. Experimental results compare the method to other coreset construction strategies and random sampling. Applications are also shown to continual learning. I think the general idea of this paper and the questions it poses are very interesting and in fact a rather underexplored area however the experimental results are quite underwhelming. The original questions posed in the paper are not really answered (can we distill a dataset?). The experiments only demonstrate that an expensive procedure that trains a model can produce synthetic images that give higher performance than a randomly selected set of images. Comments regarding experiments from Sec 4.1 The results indeed show the performance is greatly improved over using random images from the dataset, however obtaining 45% on CIFAR-10 would hardly be considered “distilling the dataset” as the full set of data yields performances of 90%+ on this task. I would be more interested to see how many samples are needed to distill the dataset such that it reaches reasonably performance. It is not completely clear why the authors didn’t do this kind of analysis but it might be due to the heavy computational cost of this method, in which case it would be good to discuss that. Similarly for MNIST the performance is quite low (effective use of regularization and can yield similar performance with a 100 images), kmeans++ based selection gives quite close results to the synthetic procedure for MNIST. I think its also a bit misleading that the authors advertise the fixed initialization performance in the abstract, this performance in my mind is mainly useful for the purposes of analyzing the method. I would also be interested to know if the coreset construction baselines can be improved. For example the kmeans++, why couldnt it be done in the feature space of a trained model (since synthetic method also trains a model). I am not an expert in the area of coresets but it would be good to know what is the state-of-the-art. A potentially interesting coreset consturction approach that also requires training a classifier can be found in this ICLR 2019 paper https://arxiv.org/pdf/1812.05159.pdf Comments regarding experiments for the continual learning application: I posted a separate comment regarding this set of experiments, which I hope the authors can address. Currently it is not 100% clear what is done. For now I will go under the assumptions that the authors are ingesting the entire dataset for each new task, using this to get their distilled data. This seems like an unfair baseline to me for the GEM setup which sees each data point only once and a for a single iteration. The authors introduce an additional step which requires multiple iterations over the whole dataset to obtain the synthetic set of images. This may be more appropriate for comparison in another continual learning setting. A few other observations/suggestions - In my understanding the label of the synthetic data is currently fixed. It would be interesting if this can also be adapted or alternatively if a fixed soft label (e.g 0.3 one class 0.7) can be assigned. - I would be interested to know if the target architecuture were different would the results change? --- Addendum: After writing this review the authors have responded to my concerns regarding the CL experiments and have begun to recompute some of the experiments using a more rigorous comparison with promising results on permuted mnist. If the CL section is clarified in the revised version (I suggest besides more decriptions in the main text to perhaps add an explicit algorithm block in the appendix) and the more rigorous comparison (single epoch to signl epoch) extend to cifar100 I will consider increasing my score as the method will be competitive for CL. I do still think the non-CL experiments (which make up the bulk of the paper are somewhat underwhelming as discussed in the review)
assistant
This paper proposes to distill a dataset into a small set of artifical images, which can be used to then train a neural network. The proposed method of doing this is based on a meta-learning flavored formulation where a GD step is done on the synthetic images and then the loss computed on the real images. The procedure appears to overfit to the initialization of the classification model and thus to resolve this the authors randomly resample the initialization throughout training. Experimental results compare the method to other coreset construction strategies and random sampling. Applications are also shown to continual learning. I think the general idea of this paper and the questions it poses are very interesting and in fact a rather underexplored area however the experimental results are quite underwhelming. The original questions posed in the paper are not really answered (can we distill a dataset?). The experiments only demonstrate that an expensive procedure that trains a model can produce synthetic images that give higher performance than a randomly selected set of images. Comments regarding experiments from Sec 4.1 The results indeed show the performance is greatly improved over using random images from the dataset, however obtaining 45% on CIFAR-10 would hardly be considered “distilling the dataset” as the full set of data yields performances of 90%+ on this task. I would be more interested to see how many samples are needed to distill the dataset such that it reaches reasonably performance. It is not completely clear why the authors didn’t do this kind of analysis but it might be due to the heavy computational cost of this method, in which case it would be good to discuss that. Similarly for MNIST the performance is quite low (effective use of regularization and can yield similar performance with a 100 images), kmeans++ based selection gives quite close results to the synthetic procedure for MNIST. I think its also a bit misleading that the authors advertise the fixed initialization performance in the abstract, this performance in my mind is mainly useful for the purposes of analyzing the method. I would also be interested to know if the coreset construction baselines can be improved. For example the kmeans++, why couldnt it be done in the feature space of a trained model (since synthetic method also trains a model). I am not an expert in the area of coresets but it would be good to know what is the state-of-the-art. A potentially interesting coreset consturction approach that also requires training a classifier can be found in this ICLR 2019 paper https://arxiv.org/pdf/1812.05159.pdf Comments regarding experiments for the continual learning application: I posted a separate comment regarding this set of experiments, which I hope the authors can address. Currently it is not 100% clear what is done. For now I will go under the assumptions that the authors are ingesting the entire dataset for each new task, using this to get their distilled data. This seems like an unfair baseline to me for the GEM setup which sees each data point only once and a for a single iteration. The authors introduce an additional step which requires multiple iterations over the whole dataset to obtain the synthetic set of images. This may be more appropriate for comparison in another continual learning setting. A few other observations/suggestions - In my understanding the label of the synthetic data is currently fixed. It would be interesting if this can also be adapted or alternatively if a fixed soft label (e.g 0.3 one class 0.7) can be assigned. - I would be interested to know if the target architecuture were different would the results change? --- Addendum: After writing this review the authors have responded to my concerns regarding the CL experiments and have begun to recompute some of the experiments using a more rigorous comparison with promising results on permuted mnist. If the CL section is clarified in the revised version (I suggest besides more decriptions in the main text to perhaps add an explicit algorithm block in the appendix) and the more rigorous comparison (single epoch to signl epoch) extend to cifar100 I will consider increasing my score as the method will be competitive for CL. I do still think the non-CL experiments (which make up the bulk of the paper are somewhat underwhelming as discussed in the review)