Dataset Distillation by Automatic Training Trajectories
<|reference_start|>Dataset Distillation by Automatic Training Trajectories: Dataset Distillation is used to create a concise, yet informative, synthetic dataset that can replace the original dataset for training purposes. Some leading methods in this domain prioritize long-range matching, involving the unrolling of training trajectories with a fixed number of steps (NS) on the synthetic dataset to align with various expert training trajectories. However, traditional long-range matching methods possess an overfitting-like problem, the fixed step size NS forces synthetic dataset to distortedly conform seen expert training trajectories, resulting in a loss of generality-especially to those from unencountered architecture. We refer to this as the Accumulated Mismatching Problem (AMP), and propose a new approach, Automatic Training Trajectories (ATT), which dynamically and adaptively adjusts trajectory length NS to address the AMP. Our method outperforms existing methods particularly in tests involving cross-architectures. Moreover, owing to its adaptive nature, it exhibits enhanced stability in the face of parameter variations.<|reference_end|>
@article{liu2024dataset,
    title={Dataset Distillation by Automatic Training Trajectories},
    author={Dai Liu, Jindong Gu, Hu Cao, Carsten Trinitis, Martin Schulz},
    journal={arXiv preprint arXiv:2407.14245},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.14245},
    primaryClass={cs.CV}
}