(cid:96)(x, θ) = (cid:96)((d, t), θ) =

(cid:107)dθ − t(cid:107)2.

1
2N

4

Algorithm 1 Dataset Distillation
Input: p(θ0): distribution of initial weights; M: the number of distilled data
Input: α: step size; n: batch size; T : the number of optimization iterations; ˜η0: initial value for ˜η
1: Initialize ˜x = {˜xi}M
2: for each training step t = 1 to T do
3:
4:
5:
6:

Get a minibatch of real training data xt = {xt,j}n
Sample a batch of initial weights θ(j)
for each sampled θ(j)

i=1 randomly, ˜η ← ˜η0

(cid:96)(˜x, θ(j)
Compute updated parameter with GD: θ(j)
0 )
Evaluate the objective function on real training data: L(j) = (cid:96)(xt, θ(j)
1 )

0 ∼ p(θ0)

1 = θ(j)

0 do

(j)
0

j=1

end for
Update ˜x ← ˜x − α∇˜x

7:
8:
9:
10: end for
Output: distilled data ˜x and optimized learning rate ˜η

(cid:80)
j L(j), and ˜η ← ˜η − α∇˜η

θ

0 − ˜η∇
(cid:80)
j L(j)

(7)

(8)

matrix after one GD step with these distilled data is

θ1 = θ0 − ˜η∇θ0(cid:96)(˜x, θ0) = θ0 − ˜η
M
DATASET DISTILLATION