title: Pandora: Towards General World Model with Natural Language Actions and
    Video States
summary: World models simulate future states of the world in response to different
  actions. They facilitate interactive content creation and provides a foundation
  for grounded, long-horizon reasoning. Current foundation models do not fully
  meet the capabilities of general world models: large language models (LLMs) are
  constrained by their reliance on language modality and their limited
  understanding of the physical world, while video models lack interactive action
  control over the world simulations. This paper makes a step towards building a
  general world model by introducing Pandora, a hybrid autoregressive-diffusion
  model that simulates world states by generating videos and allows real-time
  control with free-text actions. Pandora achieves domain generality, video
  consistency, and controllability through large-scale pretraining and
  instruction tuning. Crucially, Pandora bypasses the cost of
  training-from-scratch by integrating a pretrained LLM (7B) and a pretrained
  video model, requiring only additional lightweight finetuning. We illustrate
  extensive outputs by Pandora across diverse domains (indoor/outdoor,
  natural/urban, human/robot, 2D/3D, etc.). The results indicate great potential
  of building stronger general world models with larger-scale training.
paper_authors._id: 6670953cab84abdbee4894dc
paper_authors.name: Jiannan Xiang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-18T07:18:33.894Z
paper_authors._id: 61c4b5c871a107e9d80e33fb
paper_authors.avatarUrl: /avatars/128a2a9393b0f7921323d0d94db4ecad.svg
paper_authors.fullname: Jiannan Xiang
paper_authors.type: user
paper_authors.user: jiannanx
paper_authors._id: 6670953cab84abdbee4894dd
paper_authors.name: Guangyi Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-18T15:54:58.494Z
paper_authors._id: 6108ae87823007eaf0c7bd1e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6108ae87823007eaf0c7bd1e/dKjdx9I5waJs6oUQ0_mmT.png
paper_authors.fullname: Guangyi Liu
paper_authors.type: user
paper_authors.user: guangyil
paper_authors._id: 6670953cab84abdbee4894de
paper_authors.name: Yi Gu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-18T07:18:29.341Z
paper_authors._id: 64c2ab94e818eec6128bfeb7
paper_authors.avatarUrl: /avatars/459794e67d43a4b739ddcf90d1921b31.svg
paper_authors.fullname: Yi Gu
paper_authors.type: user
paper_authors.user: wuqing157
paper_authors._id: 6670953cab84abdbee4894df
paper_authors.name: Qiyue Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-18T15:55:13.798Z
paper_authors._id: 65a0b8de51e699b22a6da8ec
paper_authors.avatarUrl: /avatars/b133a48e5011041fa840d94e0c1060ec.svg
paper_authors.fullname: Qiyue Gao
paper_authors.type: user
paper_authors.user: BertG666
paper_authors._id: 6670953cab84abdbee4894e0
paper_authors.name: Yuting Ning
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-18T15:55:20.231Z
paper_authors._id: 65ace92f64c9b93eca5c2bce
paper_authors.avatarUrl: /avatars/9fca9d018ba751a9dba79621bf0c83f1.svg
paper_authors.fullname: Yuting Ning
paper_authors.type: user
paper_authors.user: nnnyt
paper_authors._id: 6670953cab84abdbee4894e1
paper_authors.name: Yuheng Zha
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-18T21:27:33.869Z
paper_authors._id: 6170b94f3d7d0662ac33a776
paper_authors.avatarUrl: /avatars/e8f8c49deb72e418b1ed943bfb6ecaf5.svg
paper_authors.fullname: y_zha
paper_authors.type: user
paper_authors.user: yzha
paper_authors._id: 6670953cab84abdbee4894e2
paper_authors.name: Zeyu Feng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-26T08:05:19.153Z
paper_authors._id: 647e2da4e4d52fe0e01f43af
paper_authors.avatarUrl: /avatars/76b5aeed1f3e8d2235b6012c6c96080f.svg
paper_authors.fullname: Ted Feng
paper_authors.type: user
paper_authors.user: tedfeng424
paper_authors._id: 6670953cab84abdbee4894e3
paper_authors.name: Tianhua Tao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-18T15:55:38.450Z
paper_authors._id: 61ecb698382b3b92fea7305f
paper_authors.avatarUrl: /avatars/b1bcf0d9ade12172810cb65dc9c4e4d8.svg
paper_authors.fullname: Tianhua Tao
paper_authors.type: user
paper_authors.user: Tianhua
paper_authors._id: 6670953cab84abdbee4894e4
paper_authors.name: Shibo Hao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-18T07:18:32.014Z
paper_authors._id: 660ee5df35d092e3fc2a3685
paper_authors.avatarUrl: /avatars/a7e0472fb7ea49973f74e3eea13dc964.svg
paper_authors.fullname: Shibo Hao
paper_authors.type: user
paper_authors.user: Shibo-UCSD
paper_authors._id: 6670953cab84abdbee4894e5
paper_authors.name: Yemin Shi
paper_authors._id: 6670953cab84abdbee4894e6
paper_authors.name: Zhengzhong Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-18T15:55:56.288Z
paper_authors._id: 62fbdc67c776fd8821ae3f2d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62fbdc67c776fd8821ae3f2d/cI7iAZOL40RUYluo5ZVTU.png
paper_authors.fullname: Zhengzhong Liu
paper_authors.type: user
paper_authors.user: hunterhector
paper_authors._id: 6670953cab84abdbee4894e7
paper_authors.name: Eric P. Xing
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-18T15:56:05.825Z
paper_authors._id: 64ff67722ad36636be6c4542
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/sLIrNelAWPVOy4e3oo5LB.jpeg
paper_authors.fullname: Eric Xing
paper_authors.type: user
paper_authors.user: EricX003
paper_authors._id: 6670953cab84abdbee4894e8
paper_authors.name: Zhiting Hu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-18T15:56:11.682Z
paper_authors._id: 665bfa1b0d71762b8613282d
paper_authors.avatarUrl: /avatars/edbde7b1b47032339a1ecc59f8ea8f1a.svg
paper_authors.fullname: Zhiting Hu
paper_authors.type: user
paper_authors.user: zhitinghu
paper_title: Pandora: Towards General World Model with Natural Language Actions and
    Video States
paper_summary: World models simulate future states of the world in response to different
  actions. They facilitate interactive content creation and provides a foundation
  for grounded, long-horizon reasoning. Current foundation models do not fully
  meet the capabilities of general world models: large language models (LLMs) are
  constrained by their reliance on language modality and their limited
  understanding of the physical world, while video models lack interactive action
  control over the world simulations. This paper makes a step towards building a
  general world model by introducing Pandora, a hybrid autoregressive-diffusion
  model that simulates world states by generating videos and allows real-time
  control with free-text actions. Pandora achieves domain generality, video
  consistency, and controllability through large-scale pretraining and
  instruction tuning. Crucially, Pandora bypasses the cost of
  training-from-scratch by integrating a pretrained LLM (7B) and a pretrained
  video model, requiring only additional lightweight finetuning. We illustrate
  extensive outputs by Pandora across diverse domains (indoor/outdoor,
  natural/urban, human/robot, 2D/3D, etc.). The results indicate great potential
  of building stronger general world models with larger-scale training.
paper_ai_keywords: hybrid autoregressive-diffusion model world states video simulation real-time control free-text input domain generality video consistency controllability large-scale pretraining instruction tuning pretrained LLM pretrained video model lightweight fine-tuning