I − ˜η
M

˜dT ˜d = 0,

which implies that ˜dT ˜d has full rank and M ≥ D.
Discussion. The analysis above only considers a simple case but suggests that any small number of
distilled data fail to generalize to arbitrary initial θ0. This is intuitively expected as the optimization
target (cid:96)(x, θ1) = (cid:96)(x, θ0 − ˜η∇θ0(cid:96)(˜x, θ0)) depends on the local behavior of (cid:96)(x,·) around θ0, which
can be drastically different across various initializations θ0. The lower bound M ≥ D is a quite
restricting one, considering that real datasets often have thousands to even hundreds of thousands
of dimensions (e.g., images). This analysis motivates us to focus on p(θ0) distributions that yield
similar local conditions. Section 3.5 explores several practical choices. To address the limitation of
using a single GD step, we extend our method to multiple GD steps in the next section.

3.4 MULTIPLE GRADIENT DESCENT STEPS AND MULTIPLE EPOCHS

θi+1 = θi − ˜ηi∇θi(cid:96)(˜xi, θi),
DATASET DISTILLATION