Model distillation aims to distill the knowledge of a complex model into a simpler one. In this paper, we consider an alternative formulation called dataset distillation: we keep the model fixed and instead attempt to distill the knowledge from a large training dataset into a small one. The idea is to synthesize a small number of data points that do not need to come from the correct data distribution, but will, when given to the learning algorithm as training data, approximate the model trained on the original data. For example, we show that it is possible to compress 60, 000 MNIST training images into just 10 synthetic distilled images (one per class) and achieve close to original performance with only a few steps of gradient descent, given a particular fixed network initialization. We evaluate our method in a wide range of initialization settings and with different learning objectives. Experiments on multiple datasets show the advantage of our approach compared to alternative methods in most settings.
=== Post rebuttal update ===
Thanks to the addition of better baselines, I've increased my score for this paper. While I'm still not super convinced of its potential for application, I find the idea original and worth discussing at the conference.
=== Pre-rebuttal review ===
This paper presents an approach to compress a dataset into a much smaller number of synthetic samples that are optimized to yield as good performance as possible when a given model is trained on that smaller dataset. This is done by unrolling the gradient descent procedure of training such a model to allow for gradient-based optimization of synthetic samples themselves as well as the used learning rates.
In summary, my evaluation is as follow:
*Pros*
- Pretty original problem formulation
- Generally well written paper
*Cons*
- Lack of comparison with simple baselines in basic dataset distillation setting
- Use in practical applications (domain adaptation, data poisoning) yet to be convincingly demonstrated
- Possibly a mistake in the theoretical analysis of the linear case
Indeed, I found the paper to be generally quite clear and enjoyed reading it. One minor thing I struggled a bit with is the distinction between "SG steps" and "Epochs" (I believe the former corresponds to when the synthetic samples are different between GD steps, whereas the later corresponds to the number of times the method repeatedly cycles over these samples) so I would perhaps encourage the authors to emphasize that difference. 
I also find the problem statement that proposed to be interesting and thought provoking, and the solution that's proposed seems quite appropriate and well thought out.
That said, I'm worried about the following:
- Unless I misunderstood, in the basic dataset distillation setting a comparison is never provided with training on a randomly selected subset of the training set. Presumably the results are worse, but I think these results should be in the paper. I would also argue for having another baseline, which would try to (approximately) optimize the choice of which training examples are put in the subset. A very simple approach would be to take the 200 runs already performed for the random selection and select the subset providing the best accuracy on the full training set and only report the performance of that subset (instead of the mean and std of all 200 runs). In short, this would help determine to what extent there is value to synthesizing entirely new samples. Moreover, I think a simple alternative baseline for creating synthesized samples should be considered. Specifically, I'd personally would like to know the performance of using per-class k-means clustering and training on the cluster centroids as the distilled dataset.
- While I appreciate that the authors identify potential applications and report some results on them, I think they currently fall short of convincing the reader of the potential of dataset distillation for these applications. For domain adaptation, no actual domain adaptation baseline is compared against (a good candidate would be method from Daume III 2007, at the very least). For data poisoning, I find that the assumptions for attacks are pretty strong, i.e. that a) you have access to the parameters of the pre-trained models to attack and b) that the model is doing additional updates *only* on the synthesized data. If there are reasons to think that such assumptions are reasonable, I'd at least expect the paper to motivate why that is. 
- In the analysis of the simple linear case, in Equation 7, there appears to be a mistake, specifically some missing parentheses:
d^Td( (I-\eta/M d^Td)\theta_0 + \eta/M \tilde{d}^T\tilde{t}) = d^T t
i.e. there should be parentheses right after "d^Td" and right before "=". This is from replacing \theta^* by the expression for \theta_1 in Equation 6, which is what I think Equation 7 is supposed to be doing. This possibly doesn't affect some of the conclusions taken from this section, but I'd like to see this potential mistake discussed/addressed.
That said, if the authors can sufficiently address the 3 points above, I'd be willing to increase my rating for this paper.
Finally, I have a few other more minor (nice-to-have) points:
- Having in the related work a discussion on the relationship with coreset methods would be nice
- Experiments showing how well the distilled datasets transfer to different network architectures than those used in training would be interesting? Even other ML algorithms would be quite interesting?
- "We often find that the number of distilled images required to achieve good performance is an informative indicator of the dataset diversity" => I'm not sure what in the paper actually justifies / demonstrates this statement.
- Figure 4 is presented as an "Ablation study", but an ablation study is where you remove certains parts of a model or algorithm and see what happens, which isn't the case here. I think it's better described as a hyper-parameter sensitivity study. 
- Some typos:
   * the below objective => the objective below
   * w.r.t. to => w.r.t
   * the discrete part rather => the discrete parts rather
   * necesary => necessary