VideoWorld: Exploring Knowledge Learning from Unlabeled Videos
This work explores whether a deep generative model can learn complex
knowledge solely from visual input, in contrast to the prevalent focus on
text-based models like large language models (LLMs). We develop VideoWorld, an
auto-regressive video generation model trained on unlabeled video data, and
test its knowledge acquisition abilities in video-based Go and robotic control
tasks. Our experiments reveal two key findings: (1) video-only training
provides sufficient information for learning knowledge, including rules,
reasoning and planning capabilities, and (2) the representation of visual
change is crucial for knowledge acquisition. To improve both the efficiency and
efficacy of this process, we introduce the Latent Dynamics Model (LDM) as a key
component of VideoWorld. Remarkably, VideoWorld reaches a 5-dan professional
level in the Video-GoBench with just a 300-million-parameter model, without
relying on search algorithms or reward mechanisms typical in reinforcement
learning. In robotic tasks, VideoWorld effectively learns diverse control
operations and generalizes across environments, approaching the performance of
oracle models in CALVIN and RLBench. This study opens new avenues for knowledge
acquisition from visual data, with all code, data, and models open-sourced for
further research.
678f5a57bbe3bed7b802c477
Zhongwei Ren
claimed_verified
2025-01-21T10:07:03.173Z
64e1cabf12a5504dda7e4948
/avatars/53851eddb4e1cae773f3e3607181094b.svg
rzw
user
maverickrzw
678f5a57bbe3bed7b802c478
Yunchao Wei
678f5a57bbe3bed7b802c479
Xun Guo
admin_assigned
2025-01-21T14:20:14.976Z
64ae7ddf407a5cae8579c171
/avatars/f78e8958db16f5f5603ece527951ac23.svg
Xun Guo
user
xunguohf
678f5a57bbe3bed7b802c47a
Yao Zhao
678f5a57bbe3bed7b802c47b
Bingyi Kang
admin_assigned
2025-01-21T14:19:50.039Z
647b5fef6a79fbf5e996c47c
https://cdn-avatars.huggingface.co/v1/production/uploads/647b5fef6a79fbf5e996c47c/IkSMnDsCY_CyEFCiMDuxe.jpeg
Bingyi Kang
user
bykang
678f5a57bbe3bed7b802c47c
Jiashi Feng
admin_assigned
2025-01-21T14:19:44.355Z
67298e44017b96a1d0101dc4
/avatars/1f8ed1a3e911e6a3021087b9371d284c.svg
Jiashi Feng
user
jshfeng
678f5a57bbe3bed7b802c47d
Xiaojie Jin
VideoWorld: Exploring Knowledge Learning from Unlabeled Videos
This work explores whether a deep generative model can learn complex
knowledge solely from visual input, in contrast to the prevalent focus on
text-based models like large language models (LLMs). We develop VideoWorld, an
auto-regressive video generation model trained on unlabeled video data, and
test its knowledge acquisition abilities in video-based Go and robotic control
tasks. Our experiments reveal two key findings: (1) video-only training
provides sufficient information for learning knowledge, including rules,
reasoning and planning capabilities, and (2) the representation of visual
change is crucial for knowledge acquisition. To improve both the efficiency and
efficacy of this process, we introduce the Latent Dynamics Model (LDM) as a key
component of VideoWorld. Remarkably, VideoWorld reaches a 5-dan professional
level in the Video-GoBench with just a 300-million-parameter model, without
relying on search algorithms or reward mechanisms typical in reinforcement
learning. In robotic tasks, VideoWorld effectively learns diverse control
operations and generalizes across environments, approaching the performance of
oracle models in CALVIN and RLBench. This study opens new avenues for knowledge
acquisition from visual data, with all code, data, and models open-sourced for
further research.
VideoWorld, an auto-regressive video generation model trained from visual data, acquires complex knowledge including rules, reasoning, and planning, and achieves high performance in tasks like Video-Go and robotic control without traditional reinforcement learning techniques.
auto-regressive video generation VideoWorld Latent Dynamics Model (LDM) knowledge acquisition Video-GoBench CALVIN RLBench