5 DISCUSSION

In this paper, we have presented dataset distillation for compressing the knowledge of entire training
data into a few synthetic training images. We can train a network to reach high performance with a
small number of distilled images and several gradient descent steps. Finally, we demonstrate two
extended settings including adapting pre-trained models to new datasets and performing a malicious
data-poisoning attack. In the future, we plan to extend our method to compressing large-scale visual
datasets such as ImageNet and other types of data (e.g., audio and text). Also, our current method
is sensitive to the distribution of initializations. We would like to investigate other initialization
strategies, with which dataset distillation can work well.
Acknowledgments This work was supported in part by NSF 1524817 on Advancing Visual Recogni-
tion with Feature Visualizations, NSF IIS-1633310, and Berkeley Deep Drive.

10
DATASET DISTILLATION