Controlling Text-to-Image Diffusion by Orthogonal Finetuning
Large text-to-image diffusion models have impressive capabilities in
generating photorealistic images from text prompts. How to effectively guide or
control these powerful models to perform different downstream tasks becomes an
important open problem. To tackle this challenge, we introduce a principled
finetuning method -- Orthogonal Finetuning (OFT), for adapting text-to-image
diffusion models to downstream tasks. Unlike existing methods, OFT can provably
preserve hyperspherical energy which characterizes the pairwise neuron
relationship on the unit hypersphere. We find that this property is crucial for
preserving the semantic generation ability of text-to-image diffusion models.
To improve finetuning stability, we further propose Constrained Orthogonal
Finetuning (COFT) which imposes an additional radius constraint to the
hypersphere. Specifically, we consider two important finetuning text-to-image
tasks: subject-driven generation where the goal is to generate subject-specific
images given a few images of a subject and a text prompt, and controllable
generation where the goal is to enable the model to take in additional control
signals. We empirically show that our OFT framework outperforms existing
methods in generation quality and convergence speed.
6487d8d65d164d8c7f9035e2
Zeju Qiu
6487d8d65d164d8c7f9035e3
Weiyang Liu
claimed_verified
2023-06-14T09:29:24.289Z
648905d1a15c43c791d4381f
https://cdn-avatars.huggingface.co/v1/production/uploads/648905d1a15c43c791d4381f/GpqGBzsLiMHX0gWZEz3qn.jpeg
Weiyang Liu
user
wy1iu
6487d8d65d164d8c7f9035e4
Haiwen Feng
admin_assigned
2023-06-13T10:35:02.265Z
644d2ede5c1b4e14d0c065af
/avatars/34945e0ab925d8eabc6580c71e053b11.svg
Haiwen (Haven) Feng
user
havenfeng
6487d8d65d164d8c7f9035e5
Yuxuan Xue
claimed_verified
2023-06-19T11:45:05.189Z
649004d4a3f3b05681c6c1d5
/avatars/60a5414a046b2b5dafd5cfe060f96147.svg
Yuxuan Xue
user
yxue
6487d8d65d164d8c7f9035e6
Yao Feng
6487d8d65d164d8c7f9035e7
Zhen Liu
6487d8d65d164d8c7f9035e8
Dan Zhang
6487d8d65d164d8c7f9035e9
Adrian Weller
6487d8d65d164d8c7f9035ea
Bernhard Schölkopf
Controlling Text-to-Image Diffusion by Orthogonal Finetuning
Large text-to-image diffusion models have impressive capabilities in
generating photorealistic images from text prompts. How to effectively guide or
control these powerful models to perform different downstream tasks becomes an
important open problem. To tackle this challenge, we introduce a principled
finetuning method -- Orthogonal Finetuning (OFT), for adapting text-to-image
diffusion models to downstream tasks. Unlike existing methods, OFT can provably
preserve hyperspherical energy which characterizes the pairwise neuron
relationship on the unit hypersphere. We find that this property is crucial for
preserving the semantic generation ability of text-to-image diffusion models.
To improve finetuning stability, we further propose Constrained Orthogonal
Finetuning (COFT) which imposes an additional radius constraint to the
hypersphere. Specifically, we consider two important finetuning text-to-image
tasks: subject-driven generation where the goal is to generate subject-specific
images given a few images of a subject and a text prompt, and controllable
generation where the goal is to enable the model to take in additional control
signals. We empirically show that our OFT framework outperforms existing
methods in generation quality and convergence speed.
Orthogonal Finetuning and Constrained Orthogonal Finetuning methods enhance text-to-image diffusion models by preserving hyperspherical energy and improving stability, leading to better generation quality and speed.
diffusion models Orthogonal Finetuning Constrained Orthogonal Finetuning hyperspherical energy subject-driven generation controllable generation