title: CustomVideoX: 3D Reference Attention Driven Dynamic Adaptation for
    Zero-Shot Customized Video Diffusion Transformers
summary: Customized generation has achieved significant progress in image synthesis,
  yet personalized video generation remains challenging due to temporal
  inconsistencies and quality degradation. In this paper, we introduce
  CustomVideoX, an innovative framework leveraging the video diffusion
  transformer for personalized video generation from a reference image.
  CustomVideoX capitalizes on pre-trained video networks by exclusively training
  the LoRA parameters to extract reference features, ensuring both efficiency and
  adaptability. To facilitate seamless interaction between the reference image
  and video content, we propose 3D Reference Attention, which enables direct and
  simultaneous engagement of reference image features with all video frames
  across spatial and temporal dimensions. To mitigate the excessive influence of
  reference image features and textual guidance on generated video content during
  inference, we implement the Time-Aware Reference Attention Bias (TAB) strategy,
  dynamically modulating reference bias over different time steps. Additionally,
  we introduce the Entity Region-Aware Enhancement (ERAE) module, aligning highly
  activated regions of key entity tokens with reference feature injection by
  adjusting attention bias. To thoroughly evaluate personalized video generation,
  we establish a new benchmark, VideoBench, comprising over 50 objects and 100
  prompts for extensive assessment. Experimental results show that CustomVideoX
  significantly outperforms existing methods in terms of video consistency and
  quality.
paper_authors._id: 67aae4128d478dcb4b39a097
paper_authors.name: D. She
paper_authors._id: 67aae4128d478dcb4b39a098
paper_authors.name: Mushui Liu
paper_authors._id: 67aae4128d478dcb4b39a099
paper_authors.name: Jingxuan Pang
paper_authors._id: 67aae4128d478dcb4b39a09a
paper_authors.name: Jin Wang
paper_authors._id: 67aae4128d478dcb4b39a09b
paper_authors.name: Zhen Yang
paper_authors._id: 67aae4128d478dcb4b39a09c
paper_authors.name: Wanggui He
paper_authors._id: 67aae4128d478dcb4b39a09d
paper_authors.name: Guanghao Zhang
paper_authors._id: 67aae4128d478dcb4b39a09e
paper_authors.name: Yi Wang
paper_authors._id: 67aae4128d478dcb4b39a09f
paper_authors.name: Qihan Huang
paper_authors._id: 67aae4128d478dcb4b39a0a0
paper_authors.name: Haobin Tang
paper_authors._id: 67aae4128d478dcb4b39a0a1
paper_authors.name: Yunlong Yu
paper_authors._id: 67aae4128d478dcb4b39a0a2
paper_authors.name: Siming Fu
paper_title: CustomVideoX: 3D Reference Attention Driven Dynamic Adaptation for
    Zero-Shot Customized Video Diffusion Transformers
paper_summary: Customized generation has achieved significant progress in image synthesis,
  yet personalized video generation remains challenging due to temporal
  inconsistencies and quality degradation. In this paper, we introduce
  CustomVideoX, an innovative framework leveraging the video diffusion
  transformer for personalized video generation from a reference image.
  CustomVideoX capitalizes on pre-trained video networks by exclusively training
  the LoRA parameters to extract reference features, ensuring both efficiency and
  adaptability. To facilitate seamless interaction between the reference image
  and video content, we propose 3D Reference Attention, which enables direct and
  simultaneous engagement of reference image features with all video frames
  across spatial and temporal dimensions. To mitigate the excessive influence of
  reference image features and textual guidance on generated video content during
  inference, we implement the Time-Aware Reference Attention Bias (TAB) strategy,
  dynamically modulating reference bias over different time steps. Additionally,
  we introduce the Entity Region-Aware Enhancement (ERAE) module, aligning highly
  activated regions of key entity tokens with reference feature injection by
  adjusting attention bias. To thoroughly evaluate personalized video generation,
  we establish a new benchmark, VideoBench, comprising over 50 objects and 100
  prompts for extensive assessment. Experimental results show that CustomVideoX
  significantly outperforms existing methods in terms of video consistency and
  quality.
paper_ai_summary: CustomVideoX, a framework using video diffusion transformer and pre-trained video networks, achieves improved personalized video generation by integrating reference images efficiently and dynamically adjusting attention bias.
paper_ai_keywords: video diffusion transformer pre-trained video networks LoRA parameters 3D Reference Attention Time-Aware Reference Attention Bias TAB strategy Entity Region-Aware Enhancement ERAE module VideoBench video consistency quality