Vid2World: Crafting Video Diffusion Models to Interactive World Models
World models, which predict transitions based on history observation and
action sequences, have shown great promise in improving data efficiency for
sequential decision making. However, existing world models often require
extensive domain-specific training and still produce low-fidelity, coarse
predictions, limiting their applicability in complex environments. In contrast,
video diffusion models trained on large, internet-scale datasets have
demonstrated impressive capabilities in generating high-quality videos that
capture diverse real-world dynamics. In this work, we present Vid2World, a
general approach for leveraging and transferring pre-trained video diffusion
models into interactive world models. To bridge the gap, Vid2World performs
casualization of a pre-trained video diffusion model by crafting its
architecture and training objective to enable autoregressive generation.
Furthermore, it introduces a causal action guidance mechanism to enhance action
controllability in the resulting interactive world model. Extensive experiments
in robot manipulation and game simulation domains show that our method offers a
scalable and effective approach for repurposing highly capable video diffusion
models to interactive world models.
682d2b9ba006b93dbe79a333
Siqiao Huang
admin_assigned
2025-05-21T18:04:10.245Z
66b06fe4ce2224a4d066cc0a
https://cdn-avatars.huggingface.co/v1/production/uploads/66b06fe4ce2224a4d066cc0a/m7O-cdeRQVU1yc6WVmMzW.png
knightnemo
user
knightnemo
682d2b9ba006b93dbe79a334
Jialong Wu
extracted_confirmed
2025-05-22T05:44:25.954Z
643b866bff50448bcfc7d1d1
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/q12v-BatVKimoi8q-coi-.jpeg
Jialong Wu
user
manchery
682d2b9ba006b93dbe79a335
Qixing Zhou
682d2b9ba006b93dbe79a336
Shangchen Miao
682d2b9ba006b93dbe79a337
Mingsheng Long
Vid2World: Crafting Video Diffusion Models to Interactive World Models
World models, which predict transitions based on history observation and
action sequences, have shown great promise in improving data efficiency for
sequential decision making. However, existing world models often require
extensive domain-specific training and still produce low-fidelity, coarse
predictions, limiting their applicability in complex environments. In contrast,
video diffusion models trained on large, internet-scale datasets have
demonstrated impressive capabilities in generating high-quality videos that
capture diverse real-world dynamics. In this work, we present Vid2World, a
general approach for leveraging and transferring pre-trained video diffusion
models into interactive world models. To bridge the gap, Vid2World performs
casualization of a pre-trained video diffusion model by crafting its
architecture and training objective to enable autoregressive generation.
Furthermore, it introduces a causal action guidance mechanism to enhance action
controllability in the resulting interactive world model. Extensive experiments
in robot manipulation and game simulation domains show that our method offers a
scalable and effective approach for repurposing highly capable video diffusion
models to interactive world models.
Vid2World repurposes pre-trained video diffusion models into interactive world models via causalization and action guidance, enhancing action controllability and scalability in complex environments.
world models transitions history observation action sequences data efficiency sequential decision making low-fidelity coarse predictions video diffusion models internet-scale datasets high-quality videos real-world dynamics autoregressive generation causal action guidance robot manipulation game simulation