Neural Spectral Decomposition for Dataset Distillation
<|reference_start|>Neural Spectral Decomposition for Dataset Distillation: In this paper, we propose Neural Spectrum Decomposition, a generic decomposition framework for dataset distillation. Unlike previous methods, we consider the entire dataset as a high-dimensional observation that is low-rank across all dimensions. We aim to discover the low-rank representation of the entire dataset and perform distillation efficiently. Toward this end, we learn a set of spectrum tensors and transformation matrices, which, through simple matrix multiplication, reconstruct the data distribution. Specifically, a spectrum tensor can be mapped back to the image space by a transformation matrix, and efficient information sharing during the distillation learning process is achieved through pairwise combinations of different spectrum vectors and transformation matrices. Furthermore, we integrate a trajectory matching optimization method guided by a real distribution. Our experimental results demonstrate that our approach achieves state-of-the-art performance on benchmarks, including CIFAR10, CIFAR100, Tiny Imagenet, and ImageNet Subset. Our code are available at \url{https://github.com/slyang2021/NSD}.<|reference_end|>
@article{yang2024neural,
    title={Neural Spectral Decomposition for Dataset Distillation},
    author={Shaolei Yang and Shen Cheng and Mingbo Hong and Haoqiang Fan and Xing
  Wei and Shuaicheng Liu},
    journal={arXiv preprint arXiv:2408.16236},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.16236},
    primaryClass={cs.CV}
}