title: VideoREPA: Learning Physics for Video Generation through Relational
    Alignment with Foundation Models
summary: Recent advancements in text-to-video (T2V) diffusion models have enabled
  high-fidelity and realistic video synthesis. However, current T2V models often
  struggle to generate physically plausible content due to their limited inherent
  ability to accurately understand physics. We found that while the
  representations within T2V models possess some capacity for physics
  understanding, they lag significantly behind those from recent video
  self-supervised learning methods. To this end, we propose a novel framework
  called VideoREPA, which distills physics understanding capability from video
  understanding foundation models into T2V models by aligning token-level
  relations. This closes the physics understanding gap and enable more
  physics-plausible generation. Specifically, we introduce the Token Relation
  Distillation (TRD) loss, leveraging spatio-temporal alignment to provide soft
  guidance suitable for finetuning powerful pre-trained T2V models, a critical
  departure from prior representation alignment (REPA) methods. To our knowledge,
  VideoREPA is the first REPA method designed for finetuning T2V models and
  specifically for injecting physical knowledge. Empirical evaluations show that
  VideoREPA substantially enhances the physics commonsense of baseline method,
  CogVideoX, achieving significant improvement on relevant benchmarks and
  demonstrating a strong capacity for generating videos consistent with intuitive
  physics. More video results are available at https://videorepa.github.io/.
paper_authors._id: 6842520f05049fa51eed0e9f
paper_authors.name: Xiangdong Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-06T07:41:15.290Z
paper_authors._id: 656d8d4b1f8d9b618de91369
paper_authors.avatarUrl: /avatars/884dba9e56936241034b179d11a513b9.svg
paper_authors.fullname: Xiangdong Zhang
paper_authors.type: user
paper_authors.user: aHapBean
paper_authors._id: 6842520f05049fa51eed0ea0
paper_authors.name: Jiaqi Liao
paper_authors._id: 6842520f05049fa51eed0ea1
paper_authors.name: Shaofeng Zhang
paper_authors._id: 6842520f05049fa51eed0ea2
paper_authors.name: Fanqing Meng
paper_authors._id: 6842520f05049fa51eed0ea3
paper_authors.name: Xiangpeng Wan
paper_authors._id: 6842520f05049fa51eed0ea4
paper_authors.name: Junchi Yan
paper_authors._id: 6842520f05049fa51eed0ea5
paper_authors.name: Yu Cheng
paper_title: VideoREPA: Learning Physics for Video Generation through Relational
    Alignment with Foundation Models
paper_summary: Recent advancements in text-to-video (T2V) diffusion models have enabled
  high-fidelity and realistic video synthesis. However, current T2V models often
  struggle to generate physically plausible content due to their limited inherent
  ability to accurately understand physics. We found that while the
  representations within T2V models possess some capacity for physics
  understanding, they lag significantly behind those from recent video
  self-supervised learning methods. To this end, we propose a novel framework
  called VideoREPA, which distills physics understanding capability from video
  understanding foundation models into T2V models by aligning token-level
  relations. This closes the physics understanding gap and enable more
  physics-plausible generation. Specifically, we introduce the Token Relation
  Distillation (TRD) loss, leveraging spatio-temporal alignment to provide soft
  guidance suitable for finetuning powerful pre-trained T2V models, a critical
  departure from prior representation alignment (REPA) methods. To our knowledge,
  VideoREPA is the first REPA method designed for finetuning T2V models and
  specifically for injecting physical knowledge. Empirical evaluations show that
  VideoREPA substantially enhances the physics commonsense of baseline method,
  CogVideoX, achieving significant improvement on relevant benchmarks and
  demonstrating a strong capacity for generating videos consistent with intuitive
  physics. More video results are available at https://videorepa.github.io/.
paper_ai_summary: VideoREPA enhances text-to-video synthesis by aligning token-level relations and distilling physics understanding from foundation models into T2V models.
paper_ai_keywords: T2V diffusion models physics understanding video self-supervised learning Token Relation Distillation (TRD) loss spatio-temporal alignment representation alignment (REPA) CogVideoX physics commonsense intuitive physics