  Dataset distillation aims to generate a smaller but representative subset
from a large dataset, which allows a model to be trained efficiently, meanwhile
evaluating on the original testing data distribution to achieve decent
performance. Many prior works have aimed to align with diverse aspects of the
original datasets, such as matching the training weight trajectories, gradient,
feature/BatchNorm distributions, etc. In this work, we show how to distill
various large-scale datasets such as full ImageNet-1K/21K under a conventional
input resolution of 224$\times$224 to achieve the best accuracy over all
previous approaches, including SRe$^2$L, TESLA and MTT. To achieve this, we
introduce a simple yet effective ${\bf C}$urriculum ${\bf D}$ata ${\bf
A}$ugmentation ($\texttt{CDA}$) during data synthesis that obtains the accuracy
on large-scale ImageNet-1K and 21K with 63.2% under IPC (Images Per Class) 50
and 36.1% under IPC 20, respectively. Finally, we show that, by integrating all
our enhancements together, the proposed model beats the current
state-of-the-art by more than 4% Top-1 accuracy on ImageNet-1K/21K and for the
first time, reduces the gap to its full-data training counterpart to less than
absolute 15%. Moreover, this work represents the inaugural success in dataset
distillation on larger-scale ImageNet-21K under the standard 224$\times$224
resolution. Our code and distilled ImageNet-21K dataset of 20 IPC, 2K recovery
budget are available at https://github.com/VILA-Lab/SRe2L/tree/main/CDA.

Computer Vision and Pattern Recognition;Artificial Intelligence;Machine Learning