title: Heavy Labels Out! Dataset Distillation with Label Space Lightening
summary: Dataset distillation or condensation aims to condense a large-scale training
  dataset into a much smaller synthetic one such that the training performance of
  distilled and original sets on neural networks are similar. Although the number
  of training samples can be reduced substantially, current state-of-the-art
  methods heavily rely on enormous soft labels to achieve satisfactory
  performance. As a result, the required storage can be comparable even to
  original datasets, especially for large-scale ones. To solve this problem,
  instead of storing these heavy labels, we propose a novel label-lightening
  framework termed HeLlO aiming at effective image-to-label projectors, with
  which synthetic labels can be directly generated online from synthetic images.
  Specifically, to construct such projectors, we leverage prior knowledge in
  open-source foundation models, e.g., CLIP, and introduce a LoRA-like
  fine-tuning strategy to mitigate the gap between pre-trained and target
  distributions, so that original models for soft-label generation can be
  distilled into a group of low-rank matrices. Moreover, an effective image
  optimization method is proposed to further mitigate the potential error between
  the original and distilled label generators. Extensive experiments demonstrate
  that with only about 0.003% of the original storage required for a complete set
  of soft labels, we achieve comparable performance to current state-of-the-art
  dataset distillation methods on large-scale datasets. Our code will be
  available.
paper_authors._id: 66bec6e039fcffe4062e3134
paper_authors.name: Ruonan Yu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-16T08:19:17.312Z
paper_authors._id: 6486fb33570a419f41a882e4
paper_authors.avatarUrl: /avatars/860a42074439a23c629cd23851ae4da6.svg
paper_authors.fullname: Ruonan Yu
paper_authors.type: user
paper_authors.user: roseannelexie
paper_authors._id: 66bec6e039fcffe4062e3135
paper_authors.name: Songhua Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-16T08:19:18.867Z
paper_authors._id: 64b929308b53fb5dbd059ce3
paper_authors.avatarUrl: /avatars/564c44cf45db794747a96f79f30ecd91.svg
paper_authors.fullname: Liu Songhua
paper_authors.type: user
paper_authors.user: Huage001
paper_authors._id: 66bec6e039fcffe4062e3136
paper_authors.name: Zigeng Chen
paper_authors._id: 66bec6e039fcffe4062e3137
paper_authors.name: Jingwen Ye
paper_authors._id: 66bec6e039fcffe4062e3138
paper_authors.name: Xinchao Wang
paper_title: Heavy Labels Out! Dataset Distillation with Label Space Lightening
paper_summary: Dataset distillation or condensation aims to condense a large-scale training
  dataset into a much smaller synthetic one such that the training performance of
  distilled and original sets on neural networks are similar. Although the number
  of training samples can be reduced substantially, current state-of-the-art
  methods heavily rely on enormous soft labels to achieve satisfactory
  performance. As a result, the required storage can be comparable even to
  original datasets, especially for large-scale ones. To solve this problem,
  instead of storing these heavy labels, we propose a novel label-lightening
  framework termed HeLlO aiming at effective image-to-label projectors, with
  which synthetic labels can be directly generated online from synthetic images.
  Specifically, to construct such projectors, we leverage prior knowledge in
  open-source foundation models, e.g., CLIP, and introduce a LoRA-like
  fine-tuning strategy to mitigate the gap between pre-trained and target
  distributions, so that original models for soft-label generation can be
  distilled into a group of low-rank matrices. Moreover, an effective image
  optimization method is proposed to further mitigate the potential error between
  the original and distilled label generators. Extensive experiments demonstrate
  that with only about 0.003% of the original storage required for a complete set
  of soft labels, we achieve comparable performance to current state-of-the-art
  dataset distillation methods on large-scale datasets. Our code will be
  available.
paper_ai_summary: A novel label-lightening framework, HeLlO, reduces storage requirements for dataset distillation by generating labels online, using prior knowledge from foundation models and low-rank matrices.
paper_ai_keywords: dataset distillation label-lightening HeLlO synthetic labels image-to-label projectors CLIP LoRA-like fine-tuning low-rank matrices image optimization soft-label generation