RealisDance-DiT: Simple yet Strong Baseline towards Controllable
  Character Animation in the Wild
Controllable character animation remains a challenging problem, particularly
in handling rare poses, stylized characters, character-object interactions,
complex illumination, and dynamic scenes. To tackle these issues, prior work
has largely focused on injecting pose and appearance guidance via elaborate
bypass networks, but often struggles to generalize to open-world scenarios. In
this paper, we propose a new perspective that, as long as the foundation model
is powerful enough, straightforward model modifications with flexible
fine-tuning strategies can largely address the above challenges, taking a step
towards controllable character animation in the wild. Specifically, we
introduce RealisDance-DiT, built upon the Wan-2.1 video foundation model. Our
sufficient analysis reveals that the widely adopted Reference Net design is
suboptimal for large-scale DiT models. Instead, we demonstrate that minimal
modifications to the foundation model architecture yield a surprisingly strong
baseline. We further propose the low-noise warmup and "large batches and small
iterations" strategies to accelerate model convergence during fine-tuning while
maximally preserving the priors of the foundation model. In addition, we
introduce a new test dataset that captures diverse real-world challenges,
complementing existing benchmarks such as TikTok dataset and UBC fashion video
dataset, to comprehensively evaluate the proposed method. Extensive experiments
show that RealisDance-DiT outperforms existing methods by a large margin.
68084dbc2eff5d45775d8f14
Jingkai Zhou
claimed_verified
2025-06-10T09:30:06.310Z
6434caa64b34368fdb07da48
/avatars/8da3d3d1e274ff4ad409234678b1b952.svg
Jingkai Zhou
user
theFoxofSky
68084dbc2eff5d45775d8f15
Yifan Wu
68084dbc2eff5d45775d8f16
Shikai Li
68084dbc2eff5d45775d8f17
Min Wei
68084dbc2eff5d45775d8f18
Chao Fan
68084dbc2eff5d45775d8f19
Weihua Chen
68084dbc2eff5d45775d8f1a
Wei Jiang
68084dbc2eff5d45775d8f1b
Fan Wang
RealisDance-DiT: Simple yet Strong Baseline towards Controllable
  Character Animation in the Wild
Controllable character animation remains a challenging problem, particularly
in handling rare poses, stylized characters, character-object interactions,
complex illumination, and dynamic scenes. To tackle these issues, prior work
has largely focused on injecting pose and appearance guidance via elaborate
bypass networks, but often struggles to generalize to open-world scenarios. In
this paper, we propose a new perspective that, as long as the foundation model
is powerful enough, straightforward model modifications with flexible
fine-tuning strategies can largely address the above challenges, taking a step
towards controllable character animation in the wild. Specifically, we
introduce RealisDance-DiT, built upon the Wan-2.1 video foundation model. Our
sufficient analysis reveals that the widely adopted Reference Net design is
suboptimal for large-scale DiT models. Instead, we demonstrate that minimal
modifications to the foundation model architecture yield a surprisingly strong
baseline. We further propose the low-noise warmup and "large batches and small
iterations" strategies to accelerate model convergence during fine-tuning while
maximally preserving the priors of the foundation model. In addition, we
introduce a new test dataset that captures diverse real-world challenges,
complementing existing benchmarks such as TikTok dataset and UBC fashion video
dataset, to comprehensively evaluate the proposed method. Extensive experiments
show that RealisDance-DiT outperforms existing methods by a large margin.
A modified DiT-based model, RealisDance-DiT, overcomes challenges in controllable character animation through straightforward modifications and fine-tuning strategies, outperforming existing methods on diverse datasets.
controllable character animation pose guidance appearance guidance bypass networks Reference Net DiT models foundation model fine-tuning strategies low-noise warmup model convergence test dataset TikTok dataset UBC fashion video dataset