the performance over 10 randomly sampled sets.

• Random real images: We randomly sample the same number of real images per category. We evaluate
• Optimized real images: We sample 50 sets of real images using the procedure above, pick 10 sets that
achieve the best performance on 20 held-out models and 1024 randomly chosen training images, and
evaluate the performance of these 10 sets.
• k-means: For each category, we use k-means clustering to extract the same number of cluster centroids
• Average real images: We compute the average image of all the images in each category, which is reused

as the number of distilled images in our method. We evaluate the method over 10 runs.

in different GD steps. We evaluate the model only once because average images are deterministic.

To enforce our optimized learning rate to be positive, we apply softplus to a scalar trained parameter.

14
DATASET DISTILLATION