title: RealisDance-DiT: Simple yet Strong Baseline towards Controllable
    Character Animation in the Wild
summary: Controllable character animation remains a challenging problem, particularly
  in handling rare poses, stylized characters, character-object interactions,
  complex illumination, and dynamic scenes. To tackle these issues, prior work
  has largely focused on injecting pose and appearance guidance via elaborate
  bypass networks, but often struggles to generalize to open-world scenarios. In
  this paper, we propose a new perspective that, as long as the foundation model
  is powerful enough, straightforward model modifications with flexible
  fine-tuning strategies can largely address the above challenges, taking a step
  towards controllable character animation in the wild. Specifically, we
  introduce RealisDance-DiT, built upon the Wan-2.1 video foundation model. Our
  sufficient analysis reveals that the widely adopted Reference Net design is
  suboptimal for large-scale DiT models. Instead, we demonstrate that minimal
  modifications to the foundation model architecture yield a surprisingly strong
  baseline. We further propose the low-noise warmup and "large batches and small
  iterations" strategies to accelerate model convergence during fine-tuning while
  maximally preserving the priors of the foundation model. In addition, we
  introduce a new test dataset that captures diverse real-world challenges,
  complementing existing benchmarks such as TikTok dataset and UBC fashion video
  dataset, to comprehensively evaluate the proposed method. Extensive experiments
  show that RealisDance-DiT outperforms existing methods by a large margin.
paper_authors._id: 68084dbc2eff5d45775d8f14
paper_authors.name: Jingkai Zhou
paper_authors._id: 68084dbc2eff5d45775d8f15
paper_authors.name: Yifan Wu
paper_authors._id: 68084dbc2eff5d45775d8f16
paper_authors.name: Shikai Li
paper_authors._id: 68084dbc2eff5d45775d8f17
paper_authors.name: Min Wei
paper_authors._id: 68084dbc2eff5d45775d8f18
paper_authors.name: Chao Fan
paper_authors._id: 68084dbc2eff5d45775d8f19
paper_authors.name: Weihua Chen
paper_authors._id: 68084dbc2eff5d45775d8f1a
paper_authors.name: Wei Jiang
paper_authors._id: 68084dbc2eff5d45775d8f1b
paper_authors.name: Fan Wang
paper_title: RealisDance-DiT: Simple yet Strong Baseline towards Controllable
    Character Animation in the Wild
paper_summary: Controllable character animation remains a challenging problem, particularly
  in handling rare poses, stylized characters, character-object interactions,
  complex illumination, and dynamic scenes. To tackle these issues, prior work
  has largely focused on injecting pose and appearance guidance via elaborate
  bypass networks, but often struggles to generalize to open-world scenarios. In
  this paper, we propose a new perspective that, as long as the foundation model
  is powerful enough, straightforward model modifications with flexible
  fine-tuning strategies can largely address the above challenges, taking a step
  towards controllable character animation in the wild. Specifically, we
  introduce RealisDance-DiT, built upon the Wan-2.1 video foundation model. Our
  sufficient analysis reveals that the widely adopted Reference Net design is
  suboptimal for large-scale DiT models. Instead, we demonstrate that minimal
  modifications to the foundation model architecture yield a surprisingly strong
  baseline. We further propose the low-noise warmup and "large batches and small
  iterations" strategies to accelerate model convergence during fine-tuning while
  maximally preserving the priors of the foundation model. In addition, we
  introduce a new test dataset that captures diverse real-world challenges,
  complementing existing benchmarks such as TikTok dataset and UBC fashion video
  dataset, to comprehensively evaluate the proposed method. Extensive experiments
  show that RealisDance-DiT outperforms existing methods by a large margin.
paper_ai_summary: A modified DiT-based model, RealisDance-DiT, overcomes challenges in controllable character animation through straightforward modifications and fine-tuning strategies, outperforming existing methods on diverse datasets.
paper_ai_keywords: controllable character animation pose guidance appearance guidance bypass networks Reference Net DiT models foundation model fine-tuning strategies low-noise warmup model convergence test dataset TikTok dataset UBC fashion video dataset