title: Augmenting Autotelic Agents with Large Language Models
summary: Humans learn to master open-ended repertoires of skills by imagining and
  practicing their own goals. This autotelic learning process, literally the
  pursuit of self-generated (auto) goals (telos), becomes more and more
  open-ended as the goals become more diverse, abstract and creative. The
  resulting exploration of the space of possible skills is supported by an
  inter-individual exploration: goal representations are culturally evolved and
  transmitted across individuals, in particular using language. Current
  artificial agents mostly rely on predefined goal representations corresponding
  to goal spaces that are either bounded (e.g. list of instructions), or
  unbounded (e.g. the space of possible visual inputs) but are rarely endowed
  with the ability to reshape their goal representations, to form new
  abstractions or to imagine creative goals. In this paper, we introduce a
  language model augmented autotelic agent (LMA3) that leverages a pretrained
  language model (LM) to support the representation, generation and learning of
  diverse, abstract, human-relevant goals. The LM is used as an imperfect model
  of human cultural transmission; an attempt to capture aspects of humans'
  common-sense, intuitive physics and overall interests. Specifically, it
  supports three key components of the autotelic architecture: 1)~a relabeler
  that describes the goals achieved in the agent's trajectories, 2)~a goal
  generator that suggests new high-level goals along with their decomposition
  into subgoals the agent already masters, and 3)~reward functions for each of
  these goals. Without relying on any hand-coded goal representations, reward
  functions or curriculum, we show that LMA3 agents learn to master a large
  diversity of skills in a task-agnostic text-based environment.
paper_authors._id: 646c25abdf2609a541c5067e
paper_authors.name: Cédric Colas
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-23T07:51:24.853Z
paper_authors._id: 61dc8831819c5b746f66ad68
paper_authors.avatarUrl: /avatars/25a2bc88ba74741f4579e9f41ad3cea2.svg
paper_authors.fullname: Cédric Colas
paper_authors.type: user
paper_authors.user: ccolas
paper_authors._id: 646c25abdf2609a541c5067f
paper_authors.name: Laetitia Teodorescu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-26T17:14:44.835Z
paper_authors._id: 634310806f59b79da07b3d44
paper_authors.avatarUrl: /avatars/7c229589badd39cc4f077cadf270bc74.svg
paper_authors.fullname: Laetitia
paper_authors.type: user
paper_authors.user: laetitia-teo
paper_authors._id: 646c25abdf2609a541c50680
paper_authors.name: Pierre-Yves Oudeyer
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-23T07:52:09.465Z
paper_authors._id: 60812cfb5acd42012ab4585f
paper_authors.avatarUrl: /avatars/9ff76693e026f7a72312fc2b99729d8a.svg
paper_authors.fullname: Pierre-Yves Oudeyer
paper_authors.type: user
paper_authors.user: pyoudeyer
paper_authors._id: 646c25abdf2609a541c50681
paper_authors.name: Xingdi Yuan
paper_authors._id: 646c25abdf2609a541c50682
paper_authors.name: Marc-Alexandre Côté
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-23T07:53:03.345Z
paper_authors._id: 62960afb91760de5e547e544
paper_authors.avatarUrl: /avatars/bebed5926b04d066d4c0e49d246acd78.svg
paper_authors.fullname: Marc-Alexandre Côté
paper_authors.type: user
paper_authors.user: MarcCote
paper_title: Augmenting Autotelic Agents with Large Language Models
paper_summary: Humans learn to master open-ended repertoires of skills by imagining and
  practicing their own goals. This autotelic learning process, literally the
  pursuit of self-generated (auto) goals (telos), becomes more and more
  open-ended as the goals become more diverse, abstract and creative. The
  resulting exploration of the space of possible skills is supported by an
  inter-individual exploration: goal representations are culturally evolved and
  transmitted across individuals, in particular using language. Current
  artificial agents mostly rely on predefined goal representations corresponding
  to goal spaces that are either bounded (e.g. list of instructions), or
  unbounded (e.g. the space of possible visual inputs) but are rarely endowed
  with the ability to reshape their goal representations, to form new
  abstractions or to imagine creative goals. In this paper, we introduce a
  language model augmented autotelic agent (LMA3) that leverages a pretrained
  language model (LM) to support the representation, generation and learning of
  diverse, abstract, human-relevant goals. The LM is used as an imperfect model
  of human cultural transmission; an attempt to capture aspects of humans'
  common-sense, intuitive physics and overall interests. Specifically, it
  supports three key components of the autotelic architecture: 1)~a relabeler
  that describes the goals achieved in the agent's trajectories, 2)~a goal
  generator that suggests new high-level goals along with their decomposition
  into subgoals the agent already masters, and 3)~reward functions for each of
  these goals. Without relying on any hand-coded goal representations, reward
  functions or curriculum, we show that LMA3 agents learn to master a large
  diversity of skills in a task-agnostic text-based environment.
paper_ai_summary: A language model augmented autotelic agent learns diverse, abstract goals through cultural transmission without predefined representations or curricula.
paper_ai_keywords: language model autotelic learning goal representations goal generator relabeler reward functions task-agnostic text-based environment