title: VideoWorld: Exploring Knowledge Learning from Unlabeled Videos
summary: This work explores whether a deep generative model can learn complex
  knowledge solely from visual input, in contrast to the prevalent focus on
  text-based models like large language models (LLMs). We develop VideoWorld, an
  auto-regressive video generation model trained on unlabeled video data, and
  test its knowledge acquisition abilities in video-based Go and robotic control
  tasks. Our experiments reveal two key findings: (1) video-only training
  provides sufficient information for learning knowledge, including rules,
  reasoning and planning capabilities, and (2) the representation of visual
  change is crucial for knowledge acquisition. To improve both the efficiency and
  efficacy of this process, we introduce the Latent Dynamics Model (LDM) as a key
  component of VideoWorld. Remarkably, VideoWorld reaches a 5-dan professional
  level in the Video-GoBench with just a 300-million-parameter model, without
  relying on search algorithms or reward mechanisms typical in reinforcement
  learning. In robotic tasks, VideoWorld effectively learns diverse control
  operations and generalizes across environments, approaching the performance of
  oracle models in CALVIN and RLBench. This study opens new avenues for knowledge
  acquisition from visual data, with all code, data, and models open-sourced for
  further research.
paper_authors._id: 678f5a57bbe3bed7b802c477
paper_authors.name: Zhongwei Ren
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-21T10:07:03.173Z
paper_authors._id: 64e1cabf12a5504dda7e4948
paper_authors.avatarUrl: /avatars/53851eddb4e1cae773f3e3607181094b.svg
paper_authors.fullname: rzw
paper_authors.type: user
paper_authors.user: maverickrzw
paper_authors._id: 678f5a57bbe3bed7b802c478
paper_authors.name: Yunchao Wei
paper_authors._id: 678f5a57bbe3bed7b802c479
paper_authors.name: Xun Guo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-21T14:20:14.976Z
paper_authors._id: 64ae7ddf407a5cae8579c171
paper_authors.avatarUrl: /avatars/f78e8958db16f5f5603ece527951ac23.svg
paper_authors.fullname: Xun Guo
paper_authors.type: user
paper_authors.user: xunguohf
paper_authors._id: 678f5a57bbe3bed7b802c47a
paper_authors.name: Yao Zhao
paper_authors._id: 678f5a57bbe3bed7b802c47b
paper_authors.name: Bingyi Kang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-21T14:19:50.039Z
paper_authors._id: 647b5fef6a79fbf5e996c47c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/647b5fef6a79fbf5e996c47c/IkSMnDsCY_CyEFCiMDuxe.jpeg
paper_authors.fullname: Bingyi Kang
paper_authors.type: user
paper_authors.user: bykang
paper_authors._id: 678f5a57bbe3bed7b802c47c
paper_authors.name: Jiashi Feng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-21T14:19:44.355Z
paper_authors._id: 67298e44017b96a1d0101dc4
paper_authors.avatarUrl: /avatars/1f8ed1a3e911e6a3021087b9371d284c.svg
paper_authors.fullname: Jiashi Feng
paper_authors.type: user
paper_authors.user: jshfeng
paper_authors._id: 678f5a57bbe3bed7b802c47d
paper_authors.name: Xiaojie Jin
paper_title: VideoWorld: Exploring Knowledge Learning from Unlabeled Videos
paper_summary: This work explores whether a deep generative model can learn complex
  knowledge solely from visual input, in contrast to the prevalent focus on
  text-based models like large language models (LLMs). We develop VideoWorld, an
  auto-regressive video generation model trained on unlabeled video data, and
  test its knowledge acquisition abilities in video-based Go and robotic control
  tasks. Our experiments reveal two key findings: (1) video-only training
  provides sufficient information for learning knowledge, including rules,
  reasoning and planning capabilities, and (2) the representation of visual
  change is crucial for knowledge acquisition. To improve both the efficiency and
  efficacy of this process, we introduce the Latent Dynamics Model (LDM) as a key
  component of VideoWorld. Remarkably, VideoWorld reaches a 5-dan professional
  level in the Video-GoBench with just a 300-million-parameter model, without
  relying on search algorithms or reward mechanisms typical in reinforcement
  learning. In robotic tasks, VideoWorld effectively learns diverse control
  operations and generalizes across environments, approaching the performance of
  oracle models in CALVIN and RLBench. This study opens new avenues for knowledge
  acquisition from visual data, with all code, data, and models open-sourced for
  further research.
paper_ai_summary: VideoWorld, an auto-regressive video generation model trained from visual data, acquires complex knowledge including rules, reasoning, and planning, and achieves high performance in tasks like Video-Go and robotic control without traditional reinforcement learning techniques.
paper_ai_keywords: auto-regressive video generation VideoWorld Latent Dynamics Model (LDM) knowledge acquisition Video-GoBench CALVIN RLBench