Learning to Model the World with Language
To interact with humans in the world, agents need to understand the diverse
types of language that people use, relate them to the visual world, and act
based on them. While current agents learn to execute simple language
instructions from task rewards, we aim to build agents that leverage diverse
language that conveys general knowledge, describes the state of the world,
provides interactive feedback, and more. Our key idea is that language helps
agents predict the future: what will be observed, how the world will behave,
and which situations will be rewarded. This perspective unifies language
understanding with future prediction as a powerful self-supervised learning
objective. We present Dynalang, an agent that learns a multimodal world model
that predicts future text and image representations and learns to act from
imagined model rollouts. Unlike traditional agents that use language only to
predict actions, Dynalang acquires rich language understanding by using past
language also to predict future language, video, and rewards. In addition to
learning from online interaction in an environment, Dynalang can be pretrained
on datasets of text, video, or both without actions or rewards. From using
language hints in grid worlds to navigating photorealistic scans of homes,
Dynalang utilizes diverse types of language to improve task performance,
including environment descriptions, game rules, and instructions.
64cc692aa81988d07367e89a
Jessy Lin
admin_assigned
2023-08-04T12:35:47.526Z
62ae348717d21e782c847ba7
/avatars/d4273799a59a008cf7ae45cd96bc34c6.svg
Jessy Lin
user
jessylin
64cc692aa81988d07367e89b
Yuqing Du
admin_assigned
2023-08-04T12:36:05.111Z
64b7915aa8c39dc07887e035
/avatars/271bd18eb7d4b5bae8d71674ccb3a792.svg
Yuqing Du
user
yuqingd
64cc692aa81988d07367e89c
Olivia Watkins
64cc692aa81988d07367e89d
Danijar Hafner
64cc692aa81988d07367e89e
Pieter Abbeel
64cc692aa81988d07367e89f
Dan Klein
admin_assigned
2023-08-04T12:36:46.420Z
632be88b3690fb57e70e0bf1
/avatars/74ff8f30b3662db2602495bdf493d397.svg
Dan Klein
user
danjklein
64cc692aa81988d07367e8a0
Anca Dragan
Learning to Model the World with Language
To interact with humans in the world, agents need to understand the diverse
types of language that people use, relate them to the visual world, and act
based on them. While current agents learn to execute simple language
instructions from task rewards, we aim to build agents that leverage diverse
language that conveys general knowledge, describes the state of the world,
provides interactive feedback, and more. Our key idea is that language helps
agents predict the future: what will be observed, how the world will behave,
and which situations will be rewarded. This perspective unifies language
understanding with future prediction as a powerful self-supervised learning
objective. We present Dynalang, an agent that learns a multimodal world model
that predicts future text and image representations and learns to act from
imagined model rollouts. Unlike traditional agents that use language only to
predict actions, Dynalang acquires rich language understanding by using past
language also to predict future language, video, and rewards. In addition to
learning from online interaction in an environment, Dynalang can be pretrained
on datasets of text, video, or both without actions or rewards. From using
language hints in grid worlds to navigating photorealistic scans of homes,
Dynalang utilizes diverse types of language to improve task performance,
including environment descriptions, game rules, and instructions.
Dynalang, a multimodal agent, learns to predict future text and image representations using language hints to improve task performance and enrich its understanding.
multimodal world model self-supervised learning future prediction imagined model rollouts language hints grid worlds photorealistic scans environment descriptions game rules instructions