corresponding categories: e.g., the ship image in Figure 3b. These 100 images can train randomly
initialized networks to 36.79% average test accuracy on CIFAR10. Similarly, for MNIST, the 100
distilled images shown in Figure 3a can train randomly initialized networks to 79.50% test accuracy.
Multiple gradient descent steps and multiple epochs. In Figure 3, distilled images are learned for
10 GD steps applied in 3 epochs, leading to a total of 100 images (with each step containing one
image per category). Images used in early steps tend to look noisier. However, in later steps, the
distilled images gradually look like real data and share the discriminative features for these categories.
Figure 4a shows that using more steps signiﬁcantly improves the results. Figure 4b shows a similar
but slower trend as the number of epochs increases. We observe that longer training (i.e., more
epochs) can help the model learn all the knowledge from the distilled images, but the performance
DATASET DISTILLATION