Taming generative video models for zero-shot optical flow extraction
Extracting optical flow from videos remains a core computer vision problem.
Motivated by the success of large general-purpose models, we ask whether frozen
self-supervised video models trained only for future frame prediction can be
prompted, without fine-tuning, to output flow. Prior work reading out depth or
illumination from video generators required fine-tuning, which is impractical
for flow where labels are scarce and synthetic datasets suffer from a
sim-to-real gap. Inspired by the Counterfactual World Model (CWM) paradigm,
which can obtain point-wise correspondences by injecting a small tracer
perturbation into a next-frame predictor and tracking its propagation, we
extend this idea to generative video models. We explore several popular
architectures and find that successful zero-shot flow extraction in this manner
is aided by three model properties: (1) distributional prediction of future
frames (avoiding blurry or noisy outputs); (2) factorized latents that treat
each spatio-temporal patch independently; and (3) random-access decoding that
can condition on any subset of future pixels. These properties are uniquely
present in the recent Local Random Access Sequence (LRAS) architecture.
Building on LRAS, we propose KL-tracing: a novel test-time procedure that
injects a localized perturbation into the first frame, rolls out the model one
step, and computes the Kullback-Leibler divergence between perturbed and
unperturbed predictive distributions. Without any flow-specific fine-tuning,
our method outperforms state-of-the-art models on real-world TAP-Vid DAVIS
dataset (16.6% relative improvement for endpoint error) and synthetic TAP-Vid
Kubric (4.7% relative improvement). Our results indicate that counterfactual
prompting of controllable generative video models is a scalable and effective
alternative to supervised or photometric-loss approaches for high-quality flow.
687829c0001546c83aa4f91b
Seungwoo Kim
claimed_verified
2025-07-17T08:22:33.248Z
6876bd84da0a8020fe37880b
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/E3nTXyneLmVlOqFDyRkin.png
Seungwoo (Simon) Kim
user
SeKim12
687829c0001546c83aa4f91c
Khai Loong Aw
687829c0001546c83aa4f91d
Klemen Kotar
claimed_verified
2025-07-28T08:46:32.455Z
646d3cae4220471ca0c6cb13
/avatars/30e47db03a23d49d946cc5248b28959a.svg
Klemen Kotar
user
klemenk
687829c0001546c83aa4f91e
Cristobal Eyzaguirre
687829c0001546c83aa4f91f
Wanhee Lee
687829c0001546c83aa4f920
Yunong Liu
687829c0001546c83aa4f921
Jared Watrous
687829c0001546c83aa4f922
Stefan Stojanov
687829c0001546c83aa4f923
Juan Carlos Niebles
687829c0001546c83aa4f924
Jiajun Wu
687829c0001546c83aa4f925
Daniel L. K. Yamins
Taming generative video models for zero-shot optical flow extraction
Extracting optical flow from videos remains a core computer vision problem.
Motivated by the success of large general-purpose models, we ask whether frozen
self-supervised video models trained only for future frame prediction can be
prompted, without fine-tuning, to output flow. Prior work reading out depth or
illumination from video generators required fine-tuning, which is impractical
for flow where labels are scarce and synthetic datasets suffer from a
sim-to-real gap. Inspired by the Counterfactual World Model (CWM) paradigm,
which can obtain point-wise correspondences by injecting a small tracer
perturbation into a next-frame predictor and tracking its propagation, we
extend this idea to generative video models. We explore several popular
architectures and find that successful zero-shot flow extraction in this manner
is aided by three model properties: (1) distributional prediction of future
frames (avoiding blurry or noisy outputs); (2) factorized latents that treat
each spatio-temporal patch independently; and (3) random-access decoding that
can condition on any subset of future pixels. These properties are uniquely
present in the recent Local Random Access Sequence (LRAS) architecture.
Building on LRAS, we propose KL-tracing: a novel test-time procedure that
injects a localized perturbation into the first frame, rolls out the model one
step, and computes the Kullback-Leibler divergence between perturbed and
unperturbed predictive distributions. Without any flow-specific fine-tuning,
our method outperforms state-of-the-art models on real-world TAP-Vid DAVIS
dataset (16.6% relative improvement for endpoint error) and synthetic TAP-Vid
Kubric (4.7% relative improvement). Our results indicate that counterfactual
prompting of controllable generative video models is a scalable and effective
alternative to supervised or photometric-loss approaches for high-quality flow.
Counterfactual prompting of generative video models, using KL-tracing, extracts optical flow without fine-tuning, outperforming state-of-the-art methods on real and synthetic datasets.
self-supervised video models future frame prediction Counterfactual World Model (CWM) point-wise correspondences tracer perturbation generative video models distributional prediction factorized latents random-access decoding Local Random Access Sequence (LRAS) KL-tracing Kullback-Leibler divergence optical flow TAP-Vid DAVIS TAP-Vid Kubric endpoint error