Track4Gen: Teaching Video Diffusion Models to Track Points Improves
  Video Generation
While recent foundational video generators produce visually rich output, they
still struggle with appearance drift, where objects gradually degrade or change
inconsistently across frames, breaking visual coherence. We hypothesize that
this is because there is no explicit supervision in terms of spatial tracking
at the feature level. We propose Track4Gen, a spatially aware video generator
that combines video diffusion loss with point tracking across frames, providing
enhanced spatial supervision on the diffusion features. Track4Gen merges the
video generation and point tracking tasks into a single network by making
minimal changes to existing video generation architectures. Using Stable Video
Diffusion as a backbone, Track4Gen demonstrates that it is possible to unify
video generation and point tracking, which are typically handled as separate
tasks. Our extensive evaluations show that Track4Gen effectively reduces
appearance drift, resulting in temporally stable and visually coherent video
generation. Project page: hyeonho99.github.io/track4gen
675b3e766b4e158cc439a67c
Hyeonho Jeong
claimed_verified
2024-12-16T09:42:29.562Z
656ee8008bb9f4f8d95bd8f7
/avatars/4069d70f1279d928da521211c495d638.svg
Hyeonho Jeong
user
hyeonho-jeong-video
675b3e766b4e158cc439a67d
Chun-Hao Paul Huang
675b3e766b4e158cc439a67e
Jong Chul Ye
675b3e766b4e158cc439a67f
Niloy Mitra
675b3e766b4e158cc439a680
Duygu Ceylan
admin_assigned
2024-12-18T13:43:16.387Z
632322ca2c9469d56c27e1a3
/avatars/55b12a72bf67a401bcc4fd4b4fafef80.svg
Duygu Ceylan
user
duyguceylan
Track4Gen: Teaching Video Diffusion Models to Track Points Improves
  Video Generation
While recent foundational video generators produce visually rich output, they
still struggle with appearance drift, where objects gradually degrade or change
inconsistently across frames, breaking visual coherence. We hypothesize that
this is because there is no explicit supervision in terms of spatial tracking
at the feature level. We propose Track4Gen, a spatially aware video generator
that combines video diffusion loss with point tracking across frames, providing
enhanced spatial supervision on the diffusion features. Track4Gen merges the
video generation and point tracking tasks into a single network by making
minimal changes to existing video generation architectures. Using Stable Video
Diffusion as a backbone, Track4Gen demonstrates that it is possible to unify
video generation and point tracking, which are typically handled as separate
tasks. Our extensive evaluations show that Track4Gen effectively reduces
appearance drift, resulting in temporally stable and visually coherent video
generation. Project page: hyeonho99.github.io/track4gen
Track4Gen unifies video generation and point tracking to reduce appearance drift, creating temporally stable and visually coherent videos.
video diffusion loss point tracking spatial supervision diffusion features Stable Video Diffusion appearance drift temporal stability visual coherence