Diffusion Curriculum: Synthetic-to-Real Generative Curriculum Learning
  via Image-Guided Diffusion
Low-quality or scarce data has posed significant challenges for training deep
neural networks in practice. While classical data augmentation cannot
contribute very different new data, diffusion models opens up a new door to
build self-evolving AI by generating high-quality and diverse synthetic data
through text-guided prompts. However, text-only guidance cannot control
synthetic images' proximity to the original images, resulting in
out-of-distribution data detrimental to the model performance. To overcome the
limitation, we study image guidance to achieve a spectrum of interpolations
between synthetic and real images. With stronger image guidance, the generated
images are similar to the training data but hard to learn. While with weaker
image guidance, the synthetic images will be easier for model but contribute to
a larger distribution gap with the original data. The generated full spectrum
of data enables us to build a novel "Diffusion Curriculum (DisCL)". DisCL
adjusts the image guidance level of image synthesis for each training stage: It
identifies and focuses on hard samples for the model and assesses the most
effective guidance level of synthetic images to improve hard data learning. We
apply DisCL to two challenging tasks: long-tail (LT) classification and
learning from low-quality data. It focuses on lower-guidance images of
high-quality to learn prototypical features as a warm-up of learning
higher-guidance images that might be weak on diversity or quality. Extensive
experiments showcase a gain of 2.7% and 2.1% in OOD and ID macro-accuracy when
applying DisCL to iWildCam dataset. On ImageNet-LT, DisCL improves the base
model's tail-class accuracy from 4.4% to 23.64% and leads to a 4.02%
improvement in all-class accuracy.
6711d1dbff45f0171cde152c
Yijun Liang
claimed_verified
2024-10-22T08:01:54.933Z
66720ab819bebc69b5b93685
/avatars/b2f1314d9a26f6f5eaf6cebdb0d28812.svg
Joanna Liang
user
joliang17
6711d1dbff45f0171cde152d
Shweta Bhardwaj
6711d1dbff45f0171cde152e
Tianyi Zhou
claimed_verified
2024-10-18T08:12:16.307Z
647f5af5b0e96764589f3b2a
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/VJ4cDyjp5M3V5WmI5gPIU.jpeg
Tianyi Zhou
user
zhoutianyi
Diffusion Curriculum: Synthetic-to-Real Generative Curriculum Learning
  via Image-Guided Diffusion
Low-quality or scarce data has posed significant challenges for training deep
neural networks in practice. While classical data augmentation cannot
contribute very different new data, diffusion models opens up a new door to
build self-evolving AI by generating high-quality and diverse synthetic data
through text-guided prompts. However, text-only guidance cannot control
synthetic images' proximity to the original images, resulting in
out-of-distribution data detrimental to the model performance. To overcome the
limitation, we study image guidance to achieve a spectrum of interpolations
between synthetic and real images. With stronger image guidance, the generated
images are similar to the training data but hard to learn. While with weaker
image guidance, the synthetic images will be easier for model but contribute to
a larger distribution gap with the original data. The generated full spectrum
of data enables us to build a novel "Diffusion Curriculum (DisCL)". DisCL
adjusts the image guidance level of image synthesis for each training stage: It
identifies and focuses on hard samples for the model and assesses the most
effective guidance level of synthetic images to improve hard data learning. We
apply DisCL to two challenging tasks: long-tail (LT) classification and
learning from low-quality data. It focuses on lower-guidance images of
high-quality to learn prototypical features as a warm-up of learning
higher-guidance images that might be weak on diversity or quality. Extensive
experiments showcase a gain of 2.7% and 2.1% in OOD and ID macro-accuracy when
applying DisCL to iWildCam dataset. On ImageNet-LT, DisCL improves the base
model's tail-class accuracy from 4.4% to 23.64% and leads to a 4.02%
improvement in all-class accuracy.
A novel diffusion curriculum dynamically adjusts image guidance to improve learning from low-quality or scarce data, enhancing model accuracy on both in-distribution and out-of-distribution tasks.
diffusion models text-guided prompts image guidance synthetic data Diffusion Curriculum (DisCL) long-tail classification learning from low-quality data prototypical features OOD accuracy ID accuracy