title: ControlNeXt: Powerful and Efficient Control for Image and Video
    Generation
summary: Diffusion models have demonstrated remarkable and robust abilities in both
  image and video generation. To achieve greater control over generated results,
  researchers introduce additional architectures, such as ControlNet, Adapters
  and ReferenceNet, to integrate conditioning controls. However, current
  controllable generation methods often require substantial additional
  computational resources, especially for video generation, and face challenges
  in training or exhibit weak control. In this paper, we propose ControlNeXt: a
  powerful and efficient method for controllable image and video generation. We
  first design a more straightforward and efficient architecture, replacing heavy
  additional branches with minimal additional cost compared to the base model.
  Such a concise structure also allows our method to seamlessly integrate with
  other LoRA weights, enabling style alteration without the need for additional
  training. As for training, we reduce up to 90% of learnable parameters compared
  to the alternatives. Furthermore, we propose another method called Cross
  Normalization (CN) as a replacement for Zero-Convolution' to achieve fast and
  stable training convergence. We have conducted various experiments with
  different base models across images and videos, demonstrating the robustness of
  our method.
paper_authors._id: 66bac58f4de574a32968322a
paper_authors.name: Bohao Peng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-13T07:38:16.380Z
paper_authors._id: 642ad00892094a1b74dda043
paper_authors.avatarUrl: /avatars/1ac1453a399ed57613143fe39f6913e7.svg
paper_authors.fullname: BohaoPENG
paper_authors.type: user
paper_authors.user: Pbihao
paper_authors._id: 66bac58f4de574a32968322b
paper_authors.name: Jian Wang
paper_authors._id: 66bac58f4de574a32968322c
paper_authors.name: Yuechen Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-13T07:38:18.447Z
paper_authors._id: 6418554a0956be7233a1023e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6418554a0956be7233a1023e/9EKN0GoOpcDbvBDmAQEJf.png
paper_authors.fullname: zhang yuechen
paper_authors.type: user
paper_authors.user: julianjuaner
paper_authors._id: 66bac58f4de574a32968322d
paper_authors.name: Wenbo Li
paper_authors._id: 66bac58f4de574a32968322e
paper_authors.name: Ming-Chang Yang
paper_authors._id: 66bac58f4de574a32968322f
paper_authors.name: Jiaya Jia
paper_title: ControlNeXt: Powerful and Efficient Control for Image and Video
    Generation
paper_summary: Diffusion models have demonstrated remarkable and robust abilities in both
  image and video generation. To achieve greater control over generated results,
  researchers introduce additional architectures, such as ControlNet, Adapters
  and ReferenceNet, to integrate conditioning controls. However, current
  controllable generation methods often require substantial additional
  computational resources, especially for video generation, and face challenges
  in training or exhibit weak control. In this paper, we propose ControlNeXt: a
  powerful and efficient method for controllable image and video generation. We
  first design a more straightforward and efficient architecture, replacing heavy
  additional branches with minimal additional cost compared to the base model.
  Such a concise structure also allows our method to seamlessly integrate with
  other LoRA weights, enabling style alteration without the need for additional
  training. As for training, we reduce up to 90% of learnable parameters compared
  to the alternatives. Furthermore, we propose another method called Cross
  Normalization (CN) as a replacement for Zero-Convolution' to achieve fast and
  stable training convergence. We have conducted various experiments with
  different base models across images and videos, demonstrating the robustness of
  our method.
paper_ai_summary: ControlNeXt is a computationally efficient method for controllable image and video generation with a streamlined architecture and minimal parameters, using Cross Normalization for fast training convergence.
paper_ai_keywords: ControlNet Adapters ReferenceNet ControlNeXt Zero-Convolution Cross Normalization LoRA weights style alteration robustness