Video Generation with Learned Action Prior
<|reference_start|>Video Generation with Learned Action Prior: Stochastic video generation is particularly challenging when the camera is mounted on a moving platform, as camera motion interacts with observed image pixels, creating complex spatio-temporal dynamics and making the problem partially observable. Existing methods typically address this by focusing on raw pixel-level image reconstruction without explicitly modelling camera motion dynamics. We propose a solution by considering camera motion or action as part of the observed image state, modelling both image and action within a multi-modal learning framework. We introduce three models: Video Generation with Learning Action Prior (VG-LeAP) treats the image-action pair as an augmented state generated from a single latent stochastic process and uses variational inference to learn the image-action latent prior; Causal-LeAP, which establishes a causal relationship between action and the observed image frame at time $t$, learning an action prior conditioned on the observed image states; and RAFI, which integrates the augmented image-action state concept into flow matching with diffusion generative processes, demonstrating that this action-conditioned image generation concept can be extended to other diffusion-based models. We emphasize the importance of multi-modal training in partially observable video generation problems through detailed empirical studies on our new video action dataset, RoAM.<|reference_end|>
@article{sarkar2024video,
    title={Video Generation with Learned Action Prior},
    author={Meenakshi Sarkar, Devansh Bhardwaj and Debasish Ghose},
    journal={arXiv preprint arXiv:2406.14436},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.14436},
    primaryClass={cs.CV cs.RO}
}