title: VideoJAM: Joint Appearance-Motion Representations for Enhanced Motion
    Generation in Video Models
summary: Despite tremendous recent progress, generative video models still struggle to
  capture real-world motion, dynamics, and physics. We show that this limitation
  arises from the conventional pixel reconstruction objective, which biases
  models toward appearance fidelity at the expense of motion coherence. To
  address this, we introduce VideoJAM, a novel framework that instills an
  effective motion prior to video generators, by encouraging the model to learn a
  joint appearance-motion representation. VideoJAM is composed of two
  complementary units. During training, we extend the objective to predict both
  the generated pixels and their corresponding motion from a single learned
  representation. During inference, we introduce Inner-Guidance, a mechanism that
  steers the generation toward coherent motion by leveraging the model's own
  evolving motion prediction as a dynamic guidance signal. Notably, our framework
  can be applied to any video model with minimal adaptations, requiring no
  modifications to the training data or scaling of the model. VideoJAM achieves
  state-of-the-art performance in motion coherence, surpassing highly competitive
  proprietary models while also enhancing the perceived visual quality of the
  generations. These findings emphasize that appearance and motion can be
  complementary and, when effectively integrated, enhance both the visual quality
  and the coherence of video generation. Project website:
  https://hila-chefer.github.io/videojam-paper.github.io/
paper_authors._id: 67a2ec904ea0e3138ac966f2
paper_authors.name: Hila Chefer
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-02-05T04:44:03.218Z
paper_authors._id: 6181c72cdcc1df2c9de8a4d8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1655248010394-6181c72cdcc1df2c9de8a4d8.jpeg
paper_authors.fullname: Hila Chefer
paper_authors.type: user
paper_authors.user: Hila
paper_authors._id: 67a2ec904ea0e3138ac966f3
paper_authors.name: Uriel Singer
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-05T16:53:54.046Z
paper_authors._id: 6345b71843f4f2d2ed113355
paper_authors.avatarUrl: /avatars/a497669a4c53a724c4f6ea615d1dda59.svg
paper_authors.fullname: Uriel Singer
paper_authors.type: user
paper_authors.user: urielsinger
paper_authors._id: 67a2ec904ea0e3138ac966f4
paper_authors.name: Amit Zohar
paper_authors._id: 67a2ec904ea0e3138ac966f5
paper_authors.name: Yuval Kirstain
paper_authors._id: 67a2ec904ea0e3138ac966f6
paper_authors.name: Adam Polyak
paper_authors._id: 67a2ec904ea0e3138ac966f7
paper_authors.name: Yaniv Taigman
paper_authors._id: 67a2ec904ea0e3138ac966f8
paper_authors.name: Lior Wolf
paper_authors._id: 67a2ec904ea0e3138ac966f9
paper_authors.name: Shelly Sheynin
paper_title: VideoJAM: Joint Appearance-Motion Representations for Enhanced Motion
    Generation in Video Models
paper_summary: Despite tremendous recent progress, generative video models still struggle to
  capture real-world motion, dynamics, and physics. We show that this limitation
  arises from the conventional pixel reconstruction objective, which biases
  models toward appearance fidelity at the expense of motion coherence. To
  address this, we introduce VideoJAM, a novel framework that instills an
  effective motion prior to video generators, by encouraging the model to learn a
  joint appearance-motion representation. VideoJAM is composed of two
  complementary units. During training, we extend the objective to predict both
  the generated pixels and their corresponding motion from a single learned
  representation. During inference, we introduce Inner-Guidance, a mechanism that
  steers the generation toward coherent motion by leveraging the model's own
  evolving motion prediction as a dynamic guidance signal. Notably, our framework
  can be applied to any video model with minimal adaptations, requiring no
  modifications to the training data or scaling of the model. VideoJAM achieves
  state-of-the-art performance in motion coherence, surpassing highly competitive
  proprietary models while also enhancing the perceived visual quality of the
  generations. These findings emphasize that appearance and motion can be
  complementary and, when effectively integrated, enhance both the visual quality
  and the coherence of video generation. Project website:
  https://hila-chefer.github.io/videojam-paper.github.io/
paper_ai_keywords: VideoJAM LP coherence joint appearance-motion representation Inner-Guidance motion prediction motion coherence visual quality