core-set construction (Tsang et al., 2005; Har-Peled & Kushal, 2007; Bachem et al., 2017; Sener &
Savarese, 2018) and instance selection (Olvera-López et al., 2010) methods aim to select a subset
of the entire training data, such that models trained on the subset will perform as well as the model
trained on full dataset. For example, solutions to many classical linear learning algorithms, e.g.,
Perceptron (Rosenblatt, 1957) and SVMs (Hearst et al., 1998), are weighted sums of a subset of
training examples, which can be viewed as core-sets. However, algorithms constructing these subsets
require many more training examples per category than we do, in part because their “valuable” images
have to be real, whereas our distilled images are exempt from this constraint.
Gradient-based hyperparameter optimization. Our work bears similarity with gradient-based
hyperparameter optimization techniques, which compute the gradient of hyperparameter w.r.t. the
DATASET DISTILLATION