Enhancing Detail Preservation for Customized Text-to-Image Generation: A
  Regularization-Free Approach
Recent text-to-image generation models have demonstrated impressive
capability of generating text-aligned images with high fidelity. However,
generating images of novel concept provided by the user input image is still a
challenging task. To address this problem, researchers have been exploring
various methods for customizing pre-trained text-to-image generation models.
Currently, most existing methods for customizing pre-trained text-to-image
generation models involve the use of regularization techniques to prevent
over-fitting. While regularization will ease the challenge of customization and
leads to successful content creation with respect to text guidance, it may
restrict the model capability, resulting in the loss of detailed information
and inferior performance. In this work, we propose a novel framework for
customized text-to-image generation without the use of regularization.
Specifically, our proposed framework consists of an encoder network and a novel
sampling method which can tackle the over-fitting problem without the use of
regularization. With the proposed framework, we are able to customize a
large-scale text-to-image generation model within half a minute on single GPU,
with only one image provided by the user. We demonstrate in experiments that
our proposed framework outperforms existing methods, and preserves more
fine-grained details.
646d69b61aca2e9d9c708eb8
Yufan Zhou
claimed_verified
2023-05-24T18:18:55.038Z
635c2c2a7a165601151d3f85
https://cdn-avatars.huggingface.co/v1/production/uploads/1666984965757-noauth.png
Yufan Zhou
user
YfZ
646d69b61aca2e9d9c708eb9
Ruiyi Zhang
admin_assigned
2023-07-03T09:42:53.430Z
64668f982da1abc242355cbb
/avatars/c9a248b8d70b1a8f7b78265c98690570.svg
Ruiyi Zhang
user
zhangry868
646d69b61aca2e9d9c708eba
Tong Sun
646d69b61aca2e9d9c708ebb
Jinhui Xu
Enhancing Detail Preservation for Customized Text-to-Image Generation: A
  Regularization-Free Approach
Recent text-to-image generation models have demonstrated impressive
capability of generating text-aligned images with high fidelity. However,
generating images of novel concept provided by the user input image is still a
challenging task. To address this problem, researchers have been exploring
various methods for customizing pre-trained text-to-image generation models.
Currently, most existing methods for customizing pre-trained text-to-image
generation models involve the use of regularization techniques to prevent
over-fitting. While regularization will ease the challenge of customization and
leads to successful content creation with respect to text guidance, it may
restrict the model capability, resulting in the loss of detailed information
and inferior performance. In this work, we propose a novel framework for
customized text-to-image generation without the use of regularization.
Specifically, our proposed framework consists of an encoder network and a novel
sampling method which can tackle the over-fitting problem without the use of
regularization. With the proposed framework, we are able to customize a
large-scale text-to-image generation model within half a minute on single GPU,
with only one image provided by the user. We demonstrate in experiments that
our proposed framework outperforms existing methods, and preserves more
fine-grained details.
A novel framework for customizing text-to-image generation without regularization demonstrates superior performance and fine detail preservation.
encoder network sampling method over-fitting problem text-to-image generation large-scale model single GPU user image fine-grained details