ControlNeXt: Powerful and Efficient Control for Image and Video
  Generation
Diffusion models have demonstrated remarkable and robust abilities in both
image and video generation. To achieve greater control over generated results,
researchers introduce additional architectures, such as ControlNet, Adapters
and ReferenceNet, to integrate conditioning controls. However, current
controllable generation methods often require substantial additional
computational resources, especially for video generation, and face challenges
in training or exhibit weak control. In this paper, we propose ControlNeXt: a
powerful and efficient method for controllable image and video generation. We
first design a more straightforward and efficient architecture, replacing heavy
additional branches with minimal additional cost compared to the base model.
Such a concise structure also allows our method to seamlessly integrate with
other LoRA weights, enabling style alteration without the need for additional
training. As for training, we reduce up to 90% of learnable parameters compared
to the alternatives. Furthermore, we propose another method called Cross
Normalization (CN) as a replacement for Zero-Convolution' to achieve fast and
stable training convergence. We have conducted various experiments with
different base models across images and videos, demonstrating the robustness of
our method.
66bac58f4de574a32968322a
Bohao Peng
claimed_verified
2024-08-13T07:38:16.380Z
642ad00892094a1b74dda043
/avatars/1ac1453a399ed57613143fe39f6913e7.svg
BohaoPENG
user
Pbihao
66bac58f4de574a32968322b
Jian Wang
66bac58f4de574a32968322c
Yuechen Zhang
claimed_verified
2024-08-13T07:38:18.447Z
6418554a0956be7233a1023e
https://cdn-avatars.huggingface.co/v1/production/uploads/6418554a0956be7233a1023e/9EKN0GoOpcDbvBDmAQEJf.png
zhang yuechen
user
julianjuaner
66bac58f4de574a32968322d
Wenbo Li
66bac58f4de574a32968322e
Ming-Chang Yang
66bac58f4de574a32968322f
Jiaya Jia
ControlNeXt: Powerful and Efficient Control for Image and Video
  Generation
Diffusion models have demonstrated remarkable and robust abilities in both
image and video generation. To achieve greater control over generated results,
researchers introduce additional architectures, such as ControlNet, Adapters
and ReferenceNet, to integrate conditioning controls. However, current
controllable generation methods often require substantial additional
computational resources, especially for video generation, and face challenges
in training or exhibit weak control. In this paper, we propose ControlNeXt: a
powerful and efficient method for controllable image and video generation. We
first design a more straightforward and efficient architecture, replacing heavy
additional branches with minimal additional cost compared to the base model.
Such a concise structure also allows our method to seamlessly integrate with
other LoRA weights, enabling style alteration without the need for additional
training. As for training, we reduce up to 90% of learnable parameters compared
to the alternatives. Furthermore, we propose another method called Cross
Normalization (CN) as a replacement for Zero-Convolution' to achieve fast and
stable training convergence. We have conducted various experiments with
different base models across images and videos, demonstrating the robustness of
our method.
ControlNeXt is a computationally efficient method for controllable image and video generation with a streamlined architecture and minimal parameters, using Cross Normalization for fast training convergence.
ControlNet Adapters ReferenceNet ControlNeXt Zero-Convolution Cross Normalization LoRA weights style alteration robustness