θ∗ = arg min

θ

1
N

(cid:96)(xi, θ) (cid:44) arg min

(cid:96)(x, θ),

θ

(1)

N(cid:88)

i=1

3

where for notation simplicity we overload the (cid:96)(·) notation so that (cid:96)(x, θ) represents the average error
of θ over the entire dataset. We make the mild assumption that (cid:96) is twice-differentiable, which holds
true for the majority of modern machine learning models and tasks.

3.1 OPTIMIZING DISTILLED DATA

Standard training usually applies minibatch stochastic gradient descent or its variants. At each step t,
a minibatch of training data xt = {xt,j}n

j=1 is sampled to update the current parameters as

θt+1 = θt − η∇θt(cid:96)(xt, θt),

where η is the learning rate. Such a training process often takes tens of thousands or even millions
of update steps to converge. Instead, we aim to learn a tiny set of synthetic distilled training data
˜x = {˜xi}M

i=1 with M (cid:28) N and a corresponding learning rate ˜η so that a single GD step such as
DATASET DISTILLATION