RelaCtrl: Relevance-Guided Efficient Control for Diffusion Transformers
The Diffusion Transformer plays a pivotal role in advancing text-to-image and
text-to-video generation, owing primarily to its inherent scalability. However,
existing controlled diffusion transformer methods incur significant parameter
and computational overheads and suffer from inefficient resource allocation due
to their failure to account for the varying relevance of control information
across different transformer layers. To address this, we propose the
Relevance-Guided Efficient Controllable Generation framework, RelaCtrl,
enabling efficient and resource-optimized integration of control signals into
the Diffusion Transformer. First, we evaluate the relevance of each layer in
the Diffusion Transformer to the control information by assessing the
"ControlNet Relevance Score"-i.e., the impact of skipping each control layer on
both the quality of generation and the control effectiveness during inference.
Based on the strength of the relevance, we then tailor the positioning,
parameter scale, and modeling capacity of the control layers to reduce
unnecessary parameters and redundant computations. Additionally, to further
improve efficiency, we replace the self-attention and FFN in the commonly used
copy block with the carefully designed Two-Dimensional Shuffle Mixer (TDSM),
enabling efficient implementation of both the token mixer and channel mixer.
Both qualitative and quantitative experimental results demonstrate that our
approach achieves superior performance with only 15% of the parameters and
computational complexity compared to PixArt-delta. More examples are available
at https://relactrl.github.io/RelaCtrl/.
67b7f350357c2729ac216494
Ke Cao
admin_assigned
2025-02-21T15:08:00.737Z
66e4077369d1083dd97c7cd8
/avatars/0dad41e3e2f38f89b7b21c12d673f432.svg
Ke Cao
user
kecao
67b7f350357c2729ac216495
Jing Wang
67b7f350357c2729ac216496
Ao Ma
admin_assigned
2025-02-21T15:07:53.347Z
65292034825f4bba46c78581
https://cdn-avatars.huggingface.co/v1/production/uploads/65292034825f4bba46c78581/8pkETWFhha7AUM6E37Rvl.jpeg
AoMa
user
AoMa
67b7f350357c2729ac216497
Jiasong Feng
claimed_verified
2025-07-25T08:40:04.068Z
65542ae4520d9bb9d9d21659
/avatars/c89f2e76f2f1f4e162d15f4da8280f19.svg
js
user
feng8
67b7f350357c2729ac216498
Zhanjie Zhang
admin_assigned
2025-02-21T15:07:37.282Z
640496240ab5e22719f31719
/avatars/be957a1b0e1c037f03f1439d6142e9ce.svg
Zhanjie Zhang
user
zhangzhanjay
67b7f350357c2729ac216499
Xuanhua He
admin_assigned
2025-02-21T15:07:32.005Z
64375035946fb080c6fc4551
/avatars/8dba2c8911726656d4088862a2b8fe7c.svg
Xuanhua He
user
Alexhe101
67b7f350357c2729ac21649a
Shanyuan Liu
admin_assigned
2025-02-21T15:07:26.104Z
666197e302a5e5f4377a545c
/avatars/6d6ef51cc403ffccc81265ad8adf43bc.svg
liu
user
shanyuanLiu
67b7f350357c2729ac21649b
Bo Cheng
67b7f350357c2729ac21649c
Dawei Leng
claimed_verified
2025-03-19T09:48:02.900Z
649935abbe8fd92c27ab1ed8
https://cdn-avatars.huggingface.co/v1/production/uploads/649935abbe8fd92c27ab1ed8/ueWnaZtJa-oWpzupP6FV8.png
David Leon
user
DavidLeon
67b7f350357c2729ac21649d
Yuhui Yin
67b7f350357c2729ac21649e
Jie Zhang
RelaCtrl: Relevance-Guided Efficient Control for Diffusion Transformers
The Diffusion Transformer plays a pivotal role in advancing text-to-image and
text-to-video generation, owing primarily to its inherent scalability. However,
existing controlled diffusion transformer methods incur significant parameter
and computational overheads and suffer from inefficient resource allocation due
to their failure to account for the varying relevance of control information
across different transformer layers. To address this, we propose the
Relevance-Guided Efficient Controllable Generation framework, RelaCtrl,
enabling efficient and resource-optimized integration of control signals into
the Diffusion Transformer. First, we evaluate the relevance of each layer in
the Diffusion Transformer to the control information by assessing the
"ControlNet Relevance Score"-i.e., the impact of skipping each control layer on
both the quality of generation and the control effectiveness during inference.
Based on the strength of the relevance, we then tailor the positioning,
parameter scale, and modeling capacity of the control layers to reduce
unnecessary parameters and redundant computations. Additionally, to further
improve efficiency, we replace the self-attention and FFN in the commonly used
copy block with the carefully designed Two-Dimensional Shuffle Mixer (TDSM),
enabling efficient implementation of both the token mixer and channel mixer.
Both qualitative and quantitative experimental results demonstrate that our
approach achieves superior performance with only 15% of the parameters and
computational complexity compared to PixArt-delta. More examples are available
at https://relactrl.github.io/RelaCtrl/.
The Relevance-Guided Efficient Controllable Generation framework optimizes control signal integration in the Diffusion Transformer, enhancing performance while reducing parameters and computation.
Diffusion Transformer text-to-image text-to-video parameter overhead computational overhead control information ControlNet Relevance Score self-attention FFN Two-Dimensional Shuffle Mixer token mixer channel mixer