title: Learning to Model the World with Language
summary: To interact with humans in the world, agents need to understand the diverse
  types of language that people use, relate them to the visual world, and act
  based on them. While current agents learn to execute simple language
  instructions from task rewards, we aim to build agents that leverage diverse
  language that conveys general knowledge, describes the state of the world,
  provides interactive feedback, and more. Our key idea is that language helps
  agents predict the future: what will be observed, how the world will behave,
  and which situations will be rewarded. This perspective unifies language
  understanding with future prediction as a powerful self-supervised learning
  objective. We present Dynalang, an agent that learns a multimodal world model
  that predicts future text and image representations and learns to act from
  imagined model rollouts. Unlike traditional agents that use language only to
  predict actions, Dynalang acquires rich language understanding by using past
  language also to predict future language, video, and rewards. In addition to
  learning from online interaction in an environment, Dynalang can be pretrained
  on datasets of text, video, or both without actions or rewards. From using
  language hints in grid worlds to navigating photorealistic scans of homes,
  Dynalang utilizes diverse types of language to improve task performance,
  including environment descriptions, game rules, and instructions.
paper_authors._id: 64cc692aa81988d07367e89a
paper_authors.name: Jessy Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-04T12:35:47.526Z
paper_authors._id: 62ae348717d21e782c847ba7
paper_authors.avatarUrl: /avatars/d4273799a59a008cf7ae45cd96bc34c6.svg
paper_authors.fullname: Jessy Lin
paper_authors.type: user
paper_authors.user: jessylin
paper_authors._id: 64cc692aa81988d07367e89b
paper_authors.name: Yuqing Du
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-04T12:36:05.111Z
paper_authors._id: 64b7915aa8c39dc07887e035
paper_authors.avatarUrl: /avatars/271bd18eb7d4b5bae8d71674ccb3a792.svg
paper_authors.fullname: Yuqing Du
paper_authors.type: user
paper_authors.user: yuqingd
paper_authors._id: 64cc692aa81988d07367e89c
paper_authors.name: Olivia Watkins
paper_authors._id: 64cc692aa81988d07367e89d
paper_authors.name: Danijar Hafner
paper_authors._id: 64cc692aa81988d07367e89e
paper_authors.name: Pieter Abbeel
paper_authors._id: 64cc692aa81988d07367e89f
paper_authors.name: Dan Klein
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-04T12:36:46.420Z
paper_authors._id: 632be88b3690fb57e70e0bf1
paper_authors.avatarUrl: /avatars/74ff8f30b3662db2602495bdf493d397.svg
paper_authors.fullname: Dan Klein
paper_authors.type: user
paper_authors.user: danjklein
paper_authors._id: 64cc692aa81988d07367e8a0
paper_authors.name: Anca Dragan
paper_title: Learning to Model the World with Language
paper_summary: To interact with humans in the world, agents need to understand the diverse
  types of language that people use, relate them to the visual world, and act
  based on them. While current agents learn to execute simple language
  instructions from task rewards, we aim to build agents that leverage diverse
  language that conveys general knowledge, describes the state of the world,
  provides interactive feedback, and more. Our key idea is that language helps
  agents predict the future: what will be observed, how the world will behave,
  and which situations will be rewarded. This perspective unifies language
  understanding with future prediction as a powerful self-supervised learning
  objective. We present Dynalang, an agent that learns a multimodal world model
  that predicts future text and image representations and learns to act from
  imagined model rollouts. Unlike traditional agents that use language only to
  predict actions, Dynalang acquires rich language understanding by using past
  language also to predict future language, video, and rewards. In addition to
  learning from online interaction in an environment, Dynalang can be pretrained
  on datasets of text, video, or both without actions or rewards. From using
  language hints in grid worlds to navigating photorealistic scans of homes,
  Dynalang utilizes diverse types of language to improve task performance,
  including environment descriptions, game rules, and instructions.
paper_ai_keywords: multimodal world model future prediction self-supervised learning Dynalang imagined model rollouts language hints grid worlds photorealistic scans