title: Controlling Text-to-Image Diffusion by Orthogonal Finetuning
summary: Large text-to-image diffusion models have impressive capabilities in
  generating photorealistic images from text prompts. How to effectively guide or
  control these powerful models to perform different downstream tasks becomes an
  important open problem. To tackle this challenge, we introduce a principled
  finetuning method -- Orthogonal Finetuning (OFT), for adapting text-to-image
  diffusion models to downstream tasks. Unlike existing methods, OFT can provably
  preserve hyperspherical energy which characterizes the pairwise neuron
  relationship on the unit hypersphere. We find that this property is crucial for
  preserving the semantic generation ability of text-to-image diffusion models.
  To improve finetuning stability, we further propose Constrained Orthogonal
  Finetuning (COFT) which imposes an additional radius constraint to the
  hypersphere. Specifically, we consider two important finetuning text-to-image
  tasks: subject-driven generation where the goal is to generate subject-specific
  images given a few images of a subject and a text prompt, and controllable
  generation where the goal is to enable the model to take in additional control
  signals. We empirically show that our OFT framework outperforms existing
  methods in generation quality and convergence speed.
paper_authors._id: 6487d8d65d164d8c7f9035e2
paper_authors.name: Zeju Qiu
paper_authors._id: 6487d8d65d164d8c7f9035e3
paper_authors.name: Weiyang Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-14T09:29:24.289Z
paper_authors._id: 648905d1a15c43c791d4381f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/648905d1a15c43c791d4381f/GpqGBzsLiMHX0gWZEz3qn.jpeg
paper_authors.fullname: Weiyang Liu
paper_authors.type: user
paper_authors.user: wy1iu
paper_authors._id: 6487d8d65d164d8c7f9035e4
paper_authors.name: Haiwen Feng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-13T10:35:02.265Z
paper_authors._id: 644d2ede5c1b4e14d0c065af
paper_authors.avatarUrl: /avatars/34945e0ab925d8eabc6580c71e053b11.svg
paper_authors.fullname: Haiwen (Haven) Feng
paper_authors.type: user
paper_authors.user: havenfeng
paper_authors._id: 6487d8d65d164d8c7f9035e5
paper_authors.name: Yuxuan Xue
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-19T11:45:05.189Z
paper_authors._id: 649004d4a3f3b05681c6c1d5
paper_authors.avatarUrl: /avatars/60a5414a046b2b5dafd5cfe060f96147.svg
paper_authors.fullname: Yuxuan Xue
paper_authors.type: user
paper_authors.user: yxue
paper_authors._id: 6487d8d65d164d8c7f9035e6
paper_authors.name: Yao Feng
paper_authors._id: 6487d8d65d164d8c7f9035e7
paper_authors.name: Zhen Liu
paper_authors._id: 6487d8d65d164d8c7f9035e8
paper_authors.name: Dan Zhang
paper_authors._id: 6487d8d65d164d8c7f9035e9
paper_authors.name: Adrian Weller
paper_authors._id: 6487d8d65d164d8c7f9035ea
paper_authors.name: Bernhard Schölkopf
paper_title: Controlling Text-to-Image Diffusion by Orthogonal Finetuning
paper_summary: Large text-to-image diffusion models have impressive capabilities in
  generating photorealistic images from text prompts. How to effectively guide or
  control these powerful models to perform different downstream tasks becomes an
  important open problem. To tackle this challenge, we introduce a principled
  finetuning method -- Orthogonal Finetuning (OFT), for adapting text-to-image
  diffusion models to downstream tasks. Unlike existing methods, OFT can provably
  preserve hyperspherical energy which characterizes the pairwise neuron
  relationship on the unit hypersphere. We find that this property is crucial for
  preserving the semantic generation ability of text-to-image diffusion models.
  To improve finetuning stability, we further propose Constrained Orthogonal
  Finetuning (COFT) which imposes an additional radius constraint to the
  hypersphere. Specifically, we consider two important finetuning text-to-image
  tasks: subject-driven generation where the goal is to generate subject-specific
  images given a few images of a subject and a text prompt, and controllable
  generation where the goal is to enable the model to take in additional control
  signals. We empirically show that our OFT framework outperforms existing
  methods in generation quality and convergence speed.
paper_ai_keywords: diffusion models photorealistic images text prompts Orthogonal Finetuning (OFT) hyperspherical energy pairwise neuron relationship unit hypersphere semantic generation ability Constrained Orthogonal Finetuning (COFT) radius constraint subject-driven generation controllable generation control signals generation quality convergence speed