title: Track4Gen: Teaching Video Diffusion Models to Track Points Improves
    Video Generation
summary: While recent foundational video generators produce visually rich output, they
  still struggle with appearance drift, where objects gradually degrade or change
  inconsistently across frames, breaking visual coherence. We hypothesize that
  this is because there is no explicit supervision in terms of spatial tracking
  at the feature level. We propose Track4Gen, a spatially aware video generator
  that combines video diffusion loss with point tracking across frames, providing
  enhanced spatial supervision on the diffusion features. Track4Gen merges the
  video generation and point tracking tasks into a single network by making
  minimal changes to existing video generation architectures. Using Stable Video
  Diffusion as a backbone, Track4Gen demonstrates that it is possible to unify
  video generation and point tracking, which are typically handled as separate
  tasks. Our extensive evaluations show that Track4Gen effectively reduces
  appearance drift, resulting in temporally stable and visually coherent video
  generation. Project page: hyeonho99.github.io/track4gen
paper_authors._id: 675b3e766b4e158cc439a67c
paper_authors.name: Hyeonho Jeong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-16T09:42:29.562Z
paper_authors._id: 656ee8008bb9f4f8d95bd8f7
paper_authors.avatarUrl: /avatars/4069d70f1279d928da521211c495d638.svg
paper_authors.fullname: Hyeonho Jeong
paper_authors.type: user
paper_authors.user: hyeonho-jeong-video
paper_authors._id: 675b3e766b4e158cc439a67d
paper_authors.name: Chun-Hao Paul Huang
paper_authors._id: 675b3e766b4e158cc439a67e
paper_authors.name: Jong Chul Ye
paper_authors._id: 675b3e766b4e158cc439a67f
paper_authors.name: Niloy Mitra
paper_authors._id: 675b3e766b4e158cc439a680
paper_authors.name: Duygu Ceylan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-18T13:43:16.387Z
paper_authors._id: 632322ca2c9469d56c27e1a3
paper_authors.avatarUrl: /avatars/55b12a72bf67a401bcc4fd4b4fafef80.svg
paper_authors.fullname: Duygu Ceylan
paper_authors.type: user
paper_authors.user: duyguceylan
paper_title: Track4Gen: Teaching Video Diffusion Models to Track Points Improves
    Video Generation
paper_summary: While recent foundational video generators produce visually rich output, they
  still struggle with appearance drift, where objects gradually degrade or change
  inconsistently across frames, breaking visual coherence. We hypothesize that
  this is because there is no explicit supervision in terms of spatial tracking
  at the feature level. We propose Track4Gen, a spatially aware video generator
  that combines video diffusion loss with point tracking across frames, providing
  enhanced spatial supervision on the diffusion features. Track4Gen merges the
  video generation and point tracking tasks into a single network by making
  minimal changes to existing video generation architectures. Using Stable Video
  Diffusion as a backbone, Track4Gen demonstrates that it is possible to unify
  video generation and point tracking, which are typically handled as separate
  tasks. Our extensive evaluations show that Track4Gen effectively reduces
  appearance drift, resulting in temporally stable and visually coherent video
  generation. Project page: hyeonho99.github.io/track4gen
paper_ai_summary: Track4Gen unifies video generation and point tracking to reduce appearance drift, creating temporally stable and visually coherent videos.
paper_ai_keywords: video diffusion loss point tracking spatial supervision diffusion features Stable Video Diffusion appearance drift temporal stability visual coherence