of matching activation statistics of a teacher model in knowledge distillation (Lopes et al., 2017).
Our method is also related to the theoretical concept of teaching dimension, which speciﬁes the size
of dataset necessary to teach a target model to a learner (Shinohara & Miyano, 1991; Goldman &
Kearns, 1995). However, methods (Zhu, 2013; 2015) inspired by this concept need the existence of
target models, which our method does not require.
Dataset pruning, core-set construction, and instance selection. Another way to distill knowledge
is to summarize the entire dataset by a small subset, either by only using the “valuable” data for
model training (Angelova et al., 2005; Felzenszwalb et al., 2010; Lapedriza et al., 2013) or by only
labeling the “valuable” data via active learning (Cohn et al., 1996; Tong & Koller, 2001). Similarly,
core-set construction (Tsang et al., 2005; Har-Peled & Kushal, 2007; Bachem et al., 2017; Sener &
DATASET DISTILLATION