Pandora: Towards General World Model with Natural Language Actions and
  Video States
World models simulate future states of the world in response to different
actions. They facilitate interactive content creation and provides a foundation
for grounded, long-horizon reasoning. Current foundation models do not fully
meet the capabilities of general world models: large language models (LLMs) are
constrained by their reliance on language modality and their limited
understanding of the physical world, while video models lack interactive action
control over the world simulations. This paper makes a step towards building a
general world model by introducing Pandora, a hybrid autoregressive-diffusion
model that simulates world states by generating videos and allows real-time
control with free-text actions. Pandora achieves domain generality, video
consistency, and controllability through large-scale pretraining and
instruction tuning. Crucially, Pandora bypasses the cost of
training-from-scratch by integrating a pretrained LLM (7B) and a pretrained
video model, requiring only additional lightweight finetuning. We illustrate
extensive outputs by Pandora across diverse domains (indoor/outdoor,
natural/urban, human/robot, 2D/3D, etc.). The results indicate great potential
of building stronger general world models with larger-scale training.
6670953cab84abdbee4894dc
Jiannan Xiang
claimed_verified
2024-06-18T07:18:33.894Z
61c4b5c871a107e9d80e33fb
/avatars/128a2a9393b0f7921323d0d94db4ecad.svg
Jiannan Xiang
user
jiannanx
6670953cab84abdbee4894dd
Guangyi Liu
claimed_verified
2024-06-18T15:54:58.494Z
6108ae87823007eaf0c7bd1e
https://cdn-avatars.huggingface.co/v1/production/uploads/6108ae87823007eaf0c7bd1e/dKjdx9I5waJs6oUQ0_mmT.png
Guangyi Liu
user
guangyil
6670953cab84abdbee4894de
Yi Gu
claimed_verified
2024-06-18T07:18:29.341Z
64c2ab94e818eec6128bfeb7
/avatars/459794e67d43a4b739ddcf90d1921b31.svg
Yi Gu
user
wuqing157
6670953cab84abdbee4894df
Qiyue Gao
admin_assigned
2024-06-18T15:55:13.798Z
65a0b8de51e699b22a6da8ec
/avatars/b133a48e5011041fa840d94e0c1060ec.svg
Qiyue Gao
user
BertG666
6670953cab84abdbee4894e0
Yuting Ning
admin_assigned
2024-06-18T15:55:20.231Z
65ace92f64c9b93eca5c2bce
https://cdn-avatars.huggingface.co/v1/production/uploads/65ace92f64c9b93eca5c2bce/pG0JRXH-8zEy0IoaEnMNw.jpeg
Yuting Ning
user
nnnyt
6670953cab84abdbee4894e1
Yuheng Zha
claimed_verified
2024-06-18T21:27:33.869Z
6170b94f3d7d0662ac33a776
/avatars/e8f8c49deb72e418b1ed943bfb6ecaf5.svg
y_zha
user
yzha
6670953cab84abdbee4894e2
Zeyu Feng
claimed_verified
2024-06-26T08:05:19.153Z
647e2da4e4d52fe0e01f43af
/avatars/76b5aeed1f3e8d2235b6012c6c96080f.svg
Ted Feng
user
tedfeng424
6670953cab84abdbee4894e3
Tianhua Tao
admin_assigned
2024-06-18T15:55:38.450Z
61ecb698382b3b92fea7305f
/avatars/b1bcf0d9ade12172810cb65dc9c4e4d8.svg
Tianhua Tao
user
Tianhua
6670953cab84abdbee4894e4
Shibo Hao
claimed_verified
2024-06-18T07:18:32.014Z
660ee5df35d092e3fc2a3685
/avatars/a7e0472fb7ea49973f74e3eea13dc964.svg
Shibo Hao
user
Shibo-UCSD
6670953cab84abdbee4894e5
Yemin Shi
6670953cab84abdbee4894e6
Zhengzhong Liu
admin_assigned
2024-06-18T15:55:56.288Z
62fbdc67c776fd8821ae3f2d
https://cdn-avatars.huggingface.co/v1/production/uploads/62fbdc67c776fd8821ae3f2d/cI7iAZOL40RUYluo5ZVTU.png
Zhengzhong Liu
user
hunterhector
6670953cab84abdbee4894e7
Eric P. Xing
admin_assigned
2024-06-18T15:56:05.825Z
64ff67722ad36636be6c4542
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/sLIrNelAWPVOy4e3oo5LB.jpeg
Eric Xing
user
EricX003
6670953cab84abdbee4894e8
Zhiting Hu
admin_assigned
2024-06-18T15:56:11.682Z
665bfa1b0d71762b8613282d
/avatars/edbde7b1b47032339a1ecc59f8ea8f1a.svg
Zhiting Hu
user
zhitinghu
Pandora: Towards General World Model with Natural Language Actions and
  Video States
World models simulate future states of the world in response to different
actions. They facilitate interactive content creation and provides a foundation
for grounded, long-horizon reasoning. Current foundation models do not fully
meet the capabilities of general world models: large language models (LLMs) are
constrained by their reliance on language modality and their limited
understanding of the physical world, while video models lack interactive action
control over the world simulations. This paper makes a step towards building a
general world model by introducing Pandora, a hybrid autoregressive-diffusion
model that simulates world states by generating videos and allows real-time
control with free-text actions. Pandora achieves domain generality, video
consistency, and controllability through large-scale pretraining and
instruction tuning. Crucially, Pandora bypasses the cost of
training-from-scratch by integrating a pretrained LLM (7B) and a pretrained
video model, requiring only additional lightweight finetuning. We illustrate
extensive outputs by Pandora across diverse domains (indoor/outdoor,
natural/urban, human/robot, 2D/3D, etc.). The results indicate great potential
of building stronger general world models with larger-scale training.
Pandora, a hybrid autoregressive-diffusion model, simulates world states through video generation and real-time text control, achieving domain generality, video consistency, and controllability by leveraging pretrained models.
autoregressive-diffusion model world states video generation real-time control domain generality video consistency controllability large-scale pretraining instruction tuning pretrained LLM pretrained video model