3.2 DISTILLATION FOR RANDOM INITIALIZATIONS

Unfortunately, the above distilled data optimized for a given initialization do not generalize well to
other initializations. The distilled data often look like random noise (e.g., in Figure 2a) as it encodes
the information of both training dataset x and a particular network initialization θ0. To address this
issue, we turn to calculate a small number of distilled data that can work for networks with random
initializations from a speciﬁc distribution. We formulate the optimization problem as follows:

˜x∗, ˜η∗ = arg min

Eθ0∼p(θ0)L(˜x, ˜η; θ0),

˜x,˜η

(4)
DATASET DISTILLATION