Generate Anything Anywhere in Any Scene
Text-to-image diffusion models have attracted considerable interest due to
their wide applicability across diverse fields. However, challenges persist in
creating controllable models for personalized object generation. In this paper,
we first identify the entanglement issues in existing personalized generative
models, and then propose a straightforward and efficient data augmentation
training strategy that guides the diffusion model to focus solely on object
identity. By inserting the plug-and-play adapter layers from a pre-trained
controllable diffusion model, our model obtains the ability to control the
location and size of each generated personalized object. During inference, we
propose a regionally-guided sampling technique to maintain the quality and
fidelity of the generated images. Our method achieves comparable or superior
fidelity for personalized objects, yielding a robust, versatile, and
controllable text-to-image diffusion model that is capable of generating
realistic and personalized images. Our approach demonstrates significant
potential for various applications, such as those in art, entertainment, and
advertising design.
649e4deded55df9fd2458472
Yuheng Li
admin_assigned
2023-06-30T19:53:45.610Z
63043c140547362a22a91137
/avatars/d817f4e5f65c3fab7cef2fd6a3c72632.svg
Yuheng Li
user
Yuheng
649e4deded55df9fd2458473
Haotian Liu
claimed_verified
2023-06-30T19:53:32.526Z
63898b61ec1f539adc0f4da2
https://cdn-avatars.huggingface.co/v1/production/uploads/1674248167280-63898b61ec1f539adc0f4da2.jpeg
Haotian Liu
user
liuhaotian
649e4deded55df9fd2458474
Yangming Wen
649e4deded55df9fd2458475
Yong Jae Lee
claimed_verified
2023-07-03T07:27:24.245Z
649f41ee70a478f8b36b2984
/avatars/1c9a76717a450ac4aeb62a1e823d2e4a.svg
Yong Jae Lee
user
yjlee0222
Generate Anything Anywhere in Any Scene
Text-to-image diffusion models have attracted considerable interest due to
their wide applicability across diverse fields. However, challenges persist in
creating controllable models for personalized object generation. In this paper,
we first identify the entanglement issues in existing personalized generative
models, and then propose a straightforward and efficient data augmentation
training strategy that guides the diffusion model to focus solely on object
identity. By inserting the plug-and-play adapter layers from a pre-trained
controllable diffusion model, our model obtains the ability to control the
location and size of each generated personalized object. During inference, we
propose a regionally-guided sampling technique to maintain the quality and
fidelity of the generated images. Our method achieves comparable or superior
fidelity for personalized objects, yielding a robust, versatile, and
controllable text-to-image diffusion model that is capable of generating
realistic and personalized images. Our approach demonstrates significant
potential for various applications, such as those in art, entertainment, and
advertising design.
A text-to-image diffusion model is enhanced with adapter layers and regionally-guided sampling to achieve controlled generation of personalized objects with high fidelity.
diffusion models entanglement issues data augmentation plug-and-play adapter layers regionally-guided sampling object identity object location object size personalized object generation controllable diffusion model realistic images