Generating Multi-Image Synthetic Data for Text-to-Image Customization
Customization of text-to-image models enables users to insert custom concepts
and generate the concepts in unseen settings. Existing methods either rely on
costly test-time optimization or train encoders on single-image training
datasets without multi-image supervision, leading to worse image quality. We
propose a simple approach that addresses both limitations. We first leverage
existing text-to-image models and 3D datasets to create a high-quality
Synthetic Customization Dataset (SynCD) consisting of multiple images of the
same object in different lighting, backgrounds, and poses. We then propose a
new encoder architecture based on shared attention mechanisms that better
incorporate fine-grained visual details from input images. Finally, we propose
a new inference technique that mitigates overexposure issues during inference
by normalizing the text and image guidance vectors. Through extensive
experiments, we show that our model, trained on the synthetic dataset with the
proposed encoder and inference algorithm, outperforms existing tuning-free
methods on standard customization benchmarks.
67a2fddb4044bf1c86f765a3
Nupur Kumari
admin_assigned
2025-02-05T13:37:30.825Z
62f6a894c3372328414c7021
/avatars/e8b10912355712f38f10805c31bea962.svg
Nupur Kumari
user
nupurkmr9
67a2fddb4044bf1c86f765a4
Xi Yin
67a2fddb4044bf1c86f765a5
Jun-Yan Zhu
67a2fddb4044bf1c86f765a6
Ishan Misra
67a2fddb4044bf1c86f765a7
Samaneh Azadi
Generating Multi-Image Synthetic Data for Text-to-Image Customization
Customization of text-to-image models enables users to insert custom concepts
and generate the concepts in unseen settings. Existing methods either rely on
costly test-time optimization or train encoders on single-image training
datasets without multi-image supervision, leading to worse image quality. We
propose a simple approach that addresses both limitations. We first leverage
existing text-to-image models and 3D datasets to create a high-quality
Synthetic Customization Dataset (SynCD) consisting of multiple images of the
same object in different lighting, backgrounds, and poses. We then propose a
new encoder architecture based on shared attention mechanisms that better
incorporate fine-grained visual details from input images. Finally, we propose
a new inference technique that mitigates overexposure issues during inference
by normalizing the text and image guidance vectors. Through extensive
experiments, we show that our model, trained on the synthetic dataset with the
proposed encoder and inference algorithm, outperforms existing tuning-free
methods on standard customization benchmarks.
A novel encoder and inference method improve text-to-image model customization with high-quality synthetic datasets and attention mechanisms.
Synthetic Customization Dataset (SynCD) shared attention mechanisms text-to-image models overexposure issues normalization text and image guidance vectors