SCEdit: Efficient and Controllable Image Diffusion Generation via Skip
  Connection Editing
Image diffusion models have been utilized in various tasks, such as
text-to-image generation and controllable image synthesis. Recent research has
introduced tuning methods that make subtle adjustments to the original models,
yielding promising results in specific adaptations of foundational generative
diffusion models. Rather than modifying the main backbone of the diffusion
model, we delve into the role of skip connection in U-Net and reveal that
hierarchical features aggregating long-distance information across encoder and
decoder make a significant impact on the content and quality of image
generation. Based on the observation, we propose an efficient generative tuning
framework, dubbed SCEdit, which integrates and edits Skip Connection using a
lightweight tuning module named SC-Tuner. Furthermore, the proposed framework
allows for straightforward extension to controllable image synthesis by
injecting different conditions with Controllable SC-Tuner, simplifying and
unifying the network design for multi-condition inputs. Our SCEdit
substantially reduces training parameters, memory usage, and computational
expense due to its lightweight tuners, with backward propagation only passing
to the decoder blocks. Extensive experiments conducted on text-to-image
generation and controllable image synthesis tasks demonstrate the superiority
of our method in terms of efficiency and performance. Project page:
https://scedit.github.io/
658133200be5959522cf18b3
Zeyinzi Jiang
admin_assigned
2023-12-19T10:20:43.942Z
643d278b482011f5f2bd0fae
/avatars/70b8a7ffbfa2a1c4b6f5ff5e2b96b7bf.svg
jiangzeyinzi
user
jiangzeyinzi
658133200be5959522cf18b4
Chaojie Mao
admin_assigned
2023-12-19T10:20:50.783Z
6458970cab9a44f42f620a80
/avatars/f9779b0621c931f922440fec95342444.svg
chaojie mao
user
chaojiemao
658133200be5959522cf18b5
Yulin Pan
admin_assigned
2023-12-19T10:20:57.711Z
649e707035032bf979ee0093
/avatars/ca380f3eaf2ff6d95ba2a93a289930da.svg
Yulin Pan
user
PYL2077
658133200be5959522cf18b6
Zhen Han
claimed_verified
2023-12-29T09:01:07.073Z
647ffabf28b737d7b9462eb2
/avatars/210441fc6645d08b36ad43734108f914.svg
Zhen Han
user
hanzhn
658133200be5959522cf18b7
Jingfeng Zhang
SCEdit: Efficient and Controllable Image Diffusion Generation via Skip
  Connection Editing
Image diffusion models have been utilized in various tasks, such as
text-to-image generation and controllable image synthesis. Recent research has
introduced tuning methods that make subtle adjustments to the original models,
yielding promising results in specific adaptations of foundational generative
diffusion models. Rather than modifying the main backbone of the diffusion
model, we delve into the role of skip connection in U-Net and reveal that
hierarchical features aggregating long-distance information across encoder and
decoder make a significant impact on the content and quality of image
generation. Based on the observation, we propose an efficient generative tuning
framework, dubbed SCEdit, which integrates and edits Skip Connection using a
lightweight tuning module named SC-Tuner. Furthermore, the proposed framework
allows for straightforward extension to controllable image synthesis by
injecting different conditions with Controllable SC-Tuner, simplifying and
unifying the network design for multi-condition inputs. Our SCEdit
substantially reduces training parameters, memory usage, and computational
expense due to its lightweight tuners, with backward propagation only passing
to the decoder blocks. Extensive experiments conducted on text-to-image
generation and controllable image synthesis tasks demonstrate the superiority
of our method in terms of efficiency and performance. Project page:
https://scedit.github.io/
An efficient generative tuning framework, SCEdit, leverages skip connections in U-Net to enhance image generation performance and efficiency, supporting multi-condition inputs with controllable tuning modules.
diffusion models text-to-image generation controllable image synthesis U-Net skip connections hierarchical features lightweight tuning module SC-Tuner memory usage computational expense controllable SC-Tuner decoder blocks