3.3 ANALYSIS OF A SIMPLE LINEAR CASE

This section studies our formulation in a simple linear regression problem with quadratic loss. We
derive the lower bound of the size of distilled data needed to achieve the same performance as training
on the full dataset for arbitrary initialization with one GD step. Consider a dataset x containing N
data-target pairs {(di, ti)}N
i=1, where di ∈ RD and ti ∈ R, which we represent as two matrices: an
N × D data matrix d and an N × 1 target matrix t. Given mean squared error and a D × 1 weight
matrix θ, we have

(5)
We aim to learn M synthetic data-target pairs ˜x = (˜d, ˜t), where ˜d is an M × D matrix, ˜t an M × 1
matrix (M (cid:28) N), and ˜η the learning rate, to minimize (cid:96)(x, θ0 − ˜η∇θ0(cid:96)(˜x, θ0)). The updated weight

(cid:96)(x, θ) = (cid:96)((d, t), θ) =

(cid:107)dθ − t(cid:107)2.

1
2N

4
DATASET DISTILLATION