Heavy Labels Out! Dataset Distillation with Label Space Lightening
Dataset distillation or condensation aims to condense a large-scale training
dataset into a much smaller synthetic one such that the training performance of
distilled and original sets on neural networks are similar. Although the number
of training samples can be reduced substantially, current state-of-the-art
methods heavily rely on enormous soft labels to achieve satisfactory
performance. As a result, the required storage can be comparable even to
original datasets, especially for large-scale ones. To solve this problem,
instead of storing these heavy labels, we propose a novel label-lightening
framework termed HeLlO aiming at effective image-to-label projectors, with
which synthetic labels can be directly generated online from synthetic images.
Specifically, to construct such projectors, we leverage prior knowledge in
open-source foundation models, e.g., CLIP, and introduce a LoRA-like
fine-tuning strategy to mitigate the gap between pre-trained and target
distributions, so that original models for soft-label generation can be
distilled into a group of low-rank matrices. Moreover, an effective image
optimization method is proposed to further mitigate the potential error between
the original and distilled label generators. Extensive experiments demonstrate
that with only about 0.003% of the original storage required for a complete set
of soft labels, we achieve comparable performance to current state-of-the-art
dataset distillation methods on large-scale datasets. Our code will be
available.
66bec6e039fcffe4062e3134
Ruonan Yu
claimed_verified
2024-08-16T08:19:17.312Z
6486fb33570a419f41a882e4
/avatars/860a42074439a23c629cd23851ae4da6.svg
Ruonan Yu
user
roseannelexie
66bec6e039fcffe4062e3135
Songhua Liu
claimed_verified
2024-08-16T08:19:18.867Z
64b929308b53fb5dbd059ce3
/avatars/564c44cf45db794747a96f79f30ecd91.svg
Liu Songhua
user
Huage001
66bec6e039fcffe4062e3136
Zigeng Chen
66bec6e039fcffe4062e3137
Jingwen Ye
66bec6e039fcffe4062e3138
Xinchao Wang
Heavy Labels Out! Dataset Distillation with Label Space Lightening
Dataset distillation or condensation aims to condense a large-scale training
dataset into a much smaller synthetic one such that the training performance of
distilled and original sets on neural networks are similar. Although the number
of training samples can be reduced substantially, current state-of-the-art
methods heavily rely on enormous soft labels to achieve satisfactory
performance. As a result, the required storage can be comparable even to
original datasets, especially for large-scale ones. To solve this problem,
instead of storing these heavy labels, we propose a novel label-lightening
framework termed HeLlO aiming at effective image-to-label projectors, with
which synthetic labels can be directly generated online from synthetic images.
Specifically, to construct such projectors, we leverage prior knowledge in
open-source foundation models, e.g., CLIP, and introduce a LoRA-like
fine-tuning strategy to mitigate the gap between pre-trained and target
distributions, so that original models for soft-label generation can be
distilled into a group of low-rank matrices. Moreover, an effective image
optimization method is proposed to further mitigate the potential error between
the original and distilled label generators. Extensive experiments demonstrate
that with only about 0.003% of the original storage required for a complete set
of soft labels, we achieve comparable performance to current state-of-the-art
dataset distillation methods on large-scale datasets. Our code will be
available.
A novel label-lightening framework, HeLlO, reduces storage requirements for dataset distillation by generating labels online, using prior knowledge from foundation models and low-rank matrices.
dataset distillation label-lightening HeLlO synthetic labels image-to-label projectors CLIP LoRA-like fine-tuning low-rank matrices image optimization soft-label generation