title: SCEdit: Efficient and Controllable Image Diffusion Generation via Skip
    Connection Editing
summary: Image diffusion models have been utilized in various tasks, such as
  text-to-image generation and controllable image synthesis. Recent research has
  introduced tuning methods that make subtle adjustments to the original models,
  yielding promising results in specific adaptations of foundational generative
  diffusion models. Rather than modifying the main backbone of the diffusion
  model, we delve into the role of skip connection in U-Net and reveal that
  hierarchical features aggregating long-distance information across encoder and
  decoder make a significant impact on the content and quality of image
  generation. Based on the observation, we propose an efficient generative tuning
  framework, dubbed SCEdit, which integrates and edits Skip Connection using a
  lightweight tuning module named SC-Tuner. Furthermore, the proposed framework
  allows for straightforward extension to controllable image synthesis by
  injecting different conditions with Controllable SC-Tuner, simplifying and
  unifying the network design for multi-condition inputs. Our SCEdit
  substantially reduces training parameters, memory usage, and computational
  expense due to its lightweight tuners, with backward propagation only passing
  to the decoder blocks. Extensive experiments conducted on text-to-image
  generation and controllable image synthesis tasks demonstrate the superiority
  of our method in terms of efficiency and performance. Project page:
  https://scedit.github.io/
paper_authors._id: 658133200be5959522cf18b3
paper_authors.name: Zeyinzi Jiang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-19T10:20:43.942Z
paper_authors._id: 643d278b482011f5f2bd0fae
paper_authors.avatarUrl: /avatars/70b8a7ffbfa2a1c4b6f5ff5e2b96b7bf.svg
paper_authors.fullname: jiangzeyinzi
paper_authors.type: user
paper_authors.user: jiangzeyinzi
paper_authors._id: 658133200be5959522cf18b4
paper_authors.name: Chaojie Mao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-19T10:20:50.783Z
paper_authors._id: 6458970cab9a44f42f620a80
paper_authors.avatarUrl: /avatars/f9779b0621c931f922440fec95342444.svg
paper_authors.fullname: chaojie mao
paper_authors.type: user
paper_authors.user: chaojiemao
paper_authors._id: 658133200be5959522cf18b5
paper_authors.name: Yulin Pan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-19T10:20:57.711Z
paper_authors._id: 649e707035032bf979ee0093
paper_authors.avatarUrl: /avatars/ca380f3eaf2ff6d95ba2a93a289930da.svg
paper_authors.fullname: Yulin Pan
paper_authors.type: user
paper_authors.user: PYL2077
paper_authors._id: 658133200be5959522cf18b6
paper_authors.name: Zhen Han
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-29T09:01:07.073Z
paper_authors._id: 647ffabf28b737d7b9462eb2
paper_authors.avatarUrl: /avatars/210441fc6645d08b36ad43734108f914.svg
paper_authors.fullname: Zhen Han
paper_authors.type: user
paper_authors.user: hanzhn
paper_authors._id: 658133200be5959522cf18b7
paper_authors.name: Jingfeng Zhang
paper_title: SCEdit: Efficient and Controllable Image Diffusion Generation via Skip
    Connection Editing
paper_summary: Image diffusion models have been utilized in various tasks, such as
  text-to-image generation and controllable image synthesis. Recent research has
  introduced tuning methods that make subtle adjustments to the original models,
  yielding promising results in specific adaptations of foundational generative
  diffusion models. Rather than modifying the main backbone of the diffusion
  model, we delve into the role of skip connection in U-Net and reveal that
  hierarchical features aggregating long-distance information across encoder and
  decoder make a significant impact on the content and quality of image
  generation. Based on the observation, we propose an efficient generative tuning
  framework, dubbed SCEdit, which integrates and edits Skip Connection using a
  lightweight tuning module named SC-Tuner. Furthermore, the proposed framework
  allows for straightforward extension to controllable image synthesis by
  injecting different conditions with Controllable SC-Tuner, simplifying and
  unifying the network design for multi-condition inputs. Our SCEdit
  substantially reduces training parameters, memory usage, and computational
  expense due to its lightweight tuners, with backward propagation only passing
  to the decoder blocks. Extensive experiments conducted on text-to-image
  generation and controllable image synthesis tasks demonstrate the superiority
  of our method in terms of efficiency and performance. Project page:
  https://scedit.github.io/
paper_ai_keywords: diffusion models text-to-image generation controllable image synthesis Prompt Methods skip connection U-Net hierarchical features encoder decoder generative tuning framework ALEdit cascades Skip Connection lightweight tuning module SC-Tuner Controllable SC-Tuner multi-condition inputs level stability similarity blocks training parameters memory usage computational expense backward propagation