title: Scaling Instructable Agents Across Many Simulated Worlds
summary: Building embodied AI systems that can follow arbitrary language instructions
  in any 3D environment is a key challenge for creating general AI. Accomplishing
  this goal requires learning to ground language in perception and embodied
  actions, in order to accomplish complex tasks. The Scalable, Instructable,
  Multiworld Agent (SIMA) project tackles this by training agents to follow
  free-form instructions across a diverse range of virtual 3D environments,
  including curated research environments as well as open-ended, commercial video
  games. Our goal is to develop an instructable agent that can accomplish
  anything a human can do in any simulated 3D environment. Our approach focuses
  on language-driven generality while imposing minimal assumptions. Our agents
  interact with environments in real-time using a generic, human-like interface:
  the inputs are image observations and language instructions and the outputs are
  keyboard-and-mouse actions. This general approach is challenging, but it allows
  agents to ground language across many visually complex and semantically rich
  environments while also allowing us to readily run agents in new environments.
  In this paper we describe our motivation and goal, the initial progress we have
  made, and promising preliminary results on several diverse research
  environments and a variety of commercial video games.
paper_authors._id: 661fd19136201a18e5030fa8
paper_authors.name: SIMA Team
paper_authors._id: 661fd19136201a18e5030fa9
paper_authors.name: Maria Abi Raad
paper_authors._id: 661fd19136201a18e5030faa
paper_authors.name: Arun Ahuja
paper_authors._id: 661fd19136201a18e5030fab
paper_authors.name: Catarina Barros
paper_authors._id: 661fd19136201a18e5030fac
paper_authors.name: Frederic Besse
paper_authors._id: 661fd19136201a18e5030fad
paper_authors.name: Andrew Bolt
paper_authors._id: 661fd19136201a18e5030fae
paper_authors.name: Adrian Bolton
paper_authors._id: 661fd19136201a18e5030faf
paper_authors.name: Bethanie Brownfield
paper_authors._id: 661fd19136201a18e5030fb0
paper_authors.name: Gavin Buttimore
paper_authors._id: 661fd19136201a18e5030fb1
paper_authors.name: Max Cant
paper_authors._id: 661fd19136201a18e5030fb2
paper_authors.name: Sarah Chakera
paper_authors._id: 661fd19136201a18e5030fb3
paper_authors.name: Stephanie C. Y. Chan
paper_authors._id: 661fd19136201a18e5030fb4
paper_authors.name: Jeff Clune
paper_authors._id: 661fd19136201a18e5030fb5
paper_authors.name: Adrian Collister
paper_authors._id: 661fd19136201a18e5030fb6
paper_authors.name: Vikki Copeman
paper_authors._id: 661fd19136201a18e5030fb7
paper_authors.name: Alex Cullum
paper_authors._id: 661fd19136201a18e5030fb8
paper_authors.name: Ishita Dasgupta
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T08:27:20.958Z
paper_authors._id: 64f7f2871435cc7faa39104d
paper_authors.avatarUrl: /avatars/8d42adabf6610eae5169be3ac42165e3.svg
paper_authors.fullname: Ishita Dasgupta
paper_authors.type: user
paper_authors.user: ishitadg
paper_authors._id: 661fd19136201a18e5030fb9
paper_authors.name: Dario de Cesare
paper_authors._id: 661fd19136201a18e5030fba
paper_authors.name: Julia Di Trapani
paper_authors._id: 661fd19136201a18e5030fbb
paper_authors.name: Yani Donchev
paper_authors._id: 661fd19136201a18e5030fbc
paper_authors.name: Emma Dunleavy
paper_authors._id: 661fd19136201a18e5030fbd
paper_authors.name: Martin Engelcke
paper_authors._id: 661fd19136201a18e5030fbe
paper_authors.name: Ryan Faulkner
paper_authors._id: 661fd19136201a18e5030fbf
paper_authors.name: Frankie Garcia
paper_authors._id: 661fd19136201a18e5030fc0
paper_authors.name: Charles Gbadamosi
paper_authors._id: 661fd19136201a18e5030fc1
paper_authors.name: Zhitao Gong
paper_authors._id: 661fd19136201a18e5030fc2
paper_authors.name: Lucy Gonzales
paper_authors._id: 661fd19136201a18e5030fc3
paper_authors.name: Karol Gregor
paper_authors._id: 661fd19136201a18e5030fc4
paper_authors.name: Arne Olav Hallingstad
paper_authors._id: 661fd19136201a18e5030fc5
paper_authors.name: Tim Harley
paper_authors._id: 661fd19136201a18e5030fc6
paper_authors.name: Sam Haves
paper_authors._id: 661fd19136201a18e5030fc7
paper_authors.name: Felix Hill
paper_authors._id: 661fd19136201a18e5030fc8
paper_authors.name: Ed Hirst
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-16T14:45:50.085Z
paper_authors._id: 66bf0fea094fe1e139af008a
paper_authors.avatarUrl: /avatars/5b5a817bccb9d0f94f374426bbc6f775.svg
paper_authors.fullname: Ed Hirst
paper_authors.type: user
paper_authors.user: edhirst
paper_authors._id: 661fd19136201a18e5030fc9
paper_authors.name: Drew A. Hudson
paper_authors._id: 661fd19136201a18e5030fca
paper_authors.name: Steph Hughes-Fitt
paper_authors._id: 661fd19136201a18e5030fcb
paper_authors.name: Danilo J. Rezende
paper_authors._id: 661fd19136201a18e5030fcc
paper_authors.name: Mimi Jasarevic
paper_authors._id: 661fd19136201a18e5030fcd
paper_authors.name: Laura Kampis
paper_authors._id: 661fd19136201a18e5030fce
paper_authors.name: Rosemary Ke
paper_authors._id: 661fd19136201a18e5030fcf
paper_authors.name: Thomas Keck
paper_authors._id: 661fd19136201a18e5030fd0
paper_authors.name: Junkyung Kim
paper_authors._id: 661fd19136201a18e5030fd1
paper_authors.name: Oscar Knagg
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T08:30:17.815Z
paper_authors._id: 63204ed3f7db36538ca102e8
paper_authors.avatarUrl: /avatars/c61db6e2ac29776eb552bda0b6adef58.svg
paper_authors.fullname: Oscar Knagg
paper_authors.type: user
paper_authors.user: oscarknagg
paper_authors._id: 661fd19136201a18e5030fd2
paper_authors.name: Kavya Kopparapu
paper_authors._id: 661fd19136201a18e5030fd3
paper_authors.name: Andrew Lampinen
paper_authors._id: 661fd19136201a18e5030fd4
paper_authors.name: Shane Legg
paper_authors._id: 661fd19136201a18e5030fd5
paper_authors.name: Alexander Lerchner
paper_authors._id: 661fd19136201a18e5030fd6
paper_authors.name: Marjorie Limont
paper_authors._id: 661fd19136201a18e5030fd7
paper_authors.name: Yulan Liu
paper_authors._id: 661fd19136201a18e5030fd8
paper_authors.name: Maria Loks-Thompson
paper_authors._id: 661fd19136201a18e5030fd9
paper_authors.name: Joseph Marino
paper_authors._id: 661fd19136201a18e5030fda
paper_authors.name: Kathryn Martin Cussons
paper_authors._id: 661fd19136201a18e5030fdb
paper_authors.name: Loic Matthey
paper_authors._id: 661fd19136201a18e5030fdc
paper_authors.name: Siobhan Mcloughlin
paper_authors._id: 661fd19136201a18e5030fdd
paper_authors.name: Piermaria Mendolicchio
paper_authors._id: 661fd19136201a18e5030fde
paper_authors.name: Hamza Merzic
paper_authors._id: 661fd19136201a18e5030fdf
paper_authors.name: Anna Mitenkova
paper_authors._id: 661fd19136201a18e5030fe0
paper_authors.name: Alexandre Moufarek
paper_authors._id: 661fd19136201a18e5030fe1
paper_authors.name: Valeria Oliveira
paper_authors._id: 661fd19136201a18e5030fe2
paper_authors.name: Yanko Oliveira
paper_authors._id: 661fd19136201a18e5030fe3
paper_authors.name: Hannah Openshaw
paper_authors._id: 661fd19136201a18e5030fe4
paper_authors.name: Renke Pan
paper_authors._id: 661fd19136201a18e5030fe5
paper_authors.name: Aneesh Pappu
paper_authors._id: 661fd19136201a18e5030fe6
paper_authors.name: Alex Platonov
paper_authors._id: 661fd19136201a18e5030fe7
paper_authors.name: Ollie Purkiss
paper_authors._id: 661fd19136201a18e5030fe8
paper_authors.name: David Reichert
paper_authors._id: 661fd19136201a18e5030fe9
paper_authors.name: John Reid
paper_authors._id: 661fd19136201a18e5030fea
paper_authors.name: Pierre Harvey Richemond
paper_authors._id: 661fd19136201a18e5030feb
paper_authors.name: Tyson Roberts
paper_authors._id: 661fd19136201a18e5030fec
paper_authors.name: Giles Ruscoe
paper_authors._id: 661fd19136201a18e5030fed
paper_authors.name: Jaume Sanchez Elias
paper_authors._id: 661fd19136201a18e5030fee
paper_authors.name: Tasha Sandars
paper_authors._id: 661fd19136201a18e5030fef
paper_authors.name: Daniel P. Sawyer
paper_authors._id: 661fd19136201a18e5030ff0
paper_authors.name: Tim Scholtes
paper_authors._id: 661fd19136201a18e5030ff1
paper_authors.name: Guy Simmons
paper_authors._id: 661fd19136201a18e5030ff2
paper_authors.name: Daniel Slater
paper_authors._id: 661fd19136201a18e5030ff3
paper_authors.name: Hubert Soyer
paper_authors._id: 661fd19136201a18e5030ff4
paper_authors.name: Heiko Strathmann
paper_authors._id: 661fd19136201a18e5030ff5
paper_authors.name: Peter Stys
paper_authors._id: 661fd19136201a18e5030ff6
paper_authors.name: Allison C. Tam
paper_authors._id: 661fd19136201a18e5030ff7
paper_authors.name: Denis Teplyashin
paper_authors._id: 661fd19136201a18e5030ff8
paper_authors.name: Tayfun Terzi
paper_authors._id: 661fd19136201a18e5030ff9
paper_authors.name: Davide Vercelli
paper_authors._id: 661fd19136201a18e5030ffa
paper_authors.name: Bojan Vujatovic
paper_authors._id: 661fd19136201a18e5030ffb
paper_authors.name: Marcus Wainwright
paper_authors._id: 661fd19136201a18e5030ffc
paper_authors.name: Jane X. Wang
paper_authors._id: 661fd19136201a18e5030ffd
paper_authors.name: Zhengdong Wang
paper_authors._id: 661fd19136201a18e5030ffe
paper_authors.name: Daan Wierstra
paper_authors._id: 661fd19136201a18e5030fff
paper_authors.name: Duncan Williams
paper_authors._id: 661fd19136201a18e5031000
paper_authors.name: Nathaniel Wong
paper_authors._id: 661fd19136201a18e5031001
paper_authors.name: Sarah York
paper_authors._id: 661fd19136201a18e5031002
paper_authors.name: Nick Young
paper_title: Scaling Instructable Agents Across Many Simulated Worlds
paper_summary: Building embodied AI systems that can follow arbitrary language instructions
  in any 3D environment is a key challenge for creating general AI. Accomplishing
  this goal requires learning to ground language in perception and embodied
  actions, in order to accomplish complex tasks. The Scalable, Instructable,
  Multiworld Agent (SIMA) project tackles this by training agents to follow
  free-form instructions across a diverse range of virtual 3D environments,
  including curated research environments as well as open-ended, commercial video
  games. Our goal is to develop an instructable agent that can accomplish
  anything a human can do in any simulated 3D environment. Our approach focuses
  on language-driven generality while imposing minimal assumptions. Our agents
  interact with environments in real-time using a generic, human-like interface:
  the inputs are image observations and language instructions and the outputs are
  keyboard-and-mouse actions. This general approach is challenging, but it allows
  agents to ground language across many visually complex and semantically rich
  environments while also allowing us to readily run agents in new environments.
  In this paper we describe our motivation and goal, the initial progress we have
  made, and promising preliminary results on several diverse research
  environments and a variety of commercial video games.
paper_ai_summary: Developing a scalable and adaptable embodied AI agent capable of following complex language instructions across various 3D environments using minimal assumptions.
paper_ai_keywords: embodied AI language instructions 3D environments generality image observations keyboard-and-mouse actions visual complexity semantic richness