title: Artificial Generational Intelligence: Cultural Accumulation in
    Reinforcement Learning
summary: Cultural accumulation drives the open-ended and diverse progress in
  capabilities spanning human history. It builds an expanding body of knowledge
  and skills by combining individual exploration with inter-generational
  information transmission. Despite its widespread success among humans, the
  capacity for artificial learning agents to accumulate culture remains
  under-explored. In particular, approaches to reinforcement learning typically
  strive for improvements over only a single lifetime. Generational algorithms
  that do exist fail to capture the open-ended, emergent nature of cultural
  accumulation, which allows individuals to trade-off innovation and imitation.
  Building on the previously demonstrated ability for reinforcement learning
  agents to perform social learning, we find that training setups which balance
  this with independent learning give rise to cultural accumulation. These
  accumulating agents outperform those trained for a single lifetime with the
  same cumulative experience. We explore this accumulation by constructing two
  models under two distinct notions of a generation: episodic generations, in
  which accumulation occurs via in-context learning and train-time generations,
  in which accumulation occurs via in-weights learning. In-context and in-weights
  cultural accumulation can be interpreted as analogous to knowledge and skill
  accumulation, respectively. To the best of our knowledge, this work is the
  first to present general models that achieve emergent cultural accumulation in
  reinforcement learning, opening up new avenues towards more open-ended learning
  systems, as well as presenting new opportunities for modelling human culture.
paper_authors._id: 665e92b6a62bca96e5a32905
paper_authors.name: Jonathan Cook
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-04T08:35:47.465Z
paper_authors._id: 64c900bbcbcae9f106e0eb53
paper_authors.avatarUrl: /avatars/4028efd3476980ae54f8bb707db96ec0.svg
paper_authors.fullname: Jonathan Cook
paper_authors.type: user
paper_authors.user: jcook2
paper_authors._id: 665e92b6a62bca96e5a32906
paper_authors.name: Chris Lu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-04T19:28:45.965Z
paper_authors._id: 64cbfc1d3576a06fa252736d
paper_authors.avatarUrl: /avatars/7345beafa590d5c9d5ac80cb33a9f0ab.svg
paper_authors.fullname: Chris Lu
paper_authors.type: user
paper_authors.user: chrlu
paper_authors._id: 665e92b6a62bca96e5a32907
paper_authors.name: Edward Hughes
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-04T08:36:10.715Z
paper_authors._id: 64dcad22249785efad8da97b
paper_authors.avatarUrl: /avatars/11ebe5dec1e5d2e3c7cd00ec684e2319.svg
paper_authors.fullname: Edward Hughes
paper_authors.type: user
paper_authors.user: visible-ed
paper_authors._id: 665e92b6a62bca96e5a32908
paper_authors.name: Joel Z. Leibo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-04T08:36:20.150Z
paper_authors._id: 66414cb58732bcd0762d41cf
paper_authors.avatarUrl: /avatars/0f8489364711906be6c60b850e159c03.svg
paper_authors.fullname: Joel Z Leibo
paper_authors.type: user
paper_authors.user: jzleibo
paper_authors._id: 665e92b6a62bca96e5a32909
paper_authors.name: Jakob Foerster
paper_title: Artificial Generational Intelligence: Cultural Accumulation in
    Reinforcement Learning
paper_summary: Cultural accumulation drives the open-ended and diverse progress in
  capabilities spanning human history. It builds an expanding body of knowledge
  and skills by combining individual exploration with inter-generational
  information transmission. Despite its widespread success among humans, the
  capacity for artificial learning agents to accumulate culture remains
  under-explored. In particular, approaches to reinforcement learning typically
  strive for improvements over only a single lifetime. Generational algorithms
  that do exist fail to capture the open-ended, emergent nature of cultural
  accumulation, which allows individuals to trade-off innovation and imitation.
  Building on the previously demonstrated ability for reinforcement learning
  agents to perform social learning, we find that training setups which balance
  this with independent learning give rise to cultural accumulation. These
  accumulating agents outperform those trained for a single lifetime with the
  same cumulative experience. We explore this accumulation by constructing two
  models under two distinct notions of a generation: episodic generations, in
  which accumulation occurs via in-context learning and train-time generations,
  in which accumulation occurs via in-weights learning. In-context and in-weights
  cultural accumulation can be interpreted as analogous to knowledge and skill
  accumulation, respectively. To the best of our knowledge, this work is the
  first to present general models that achieve emergent cultural accumulation in
  reinforcement learning, opening up new avenues towards more open-ended learning
  systems, as well as presenting new opportunities for modelling human culture.
paper_ai_summary: Reinforcement learning agents can accumulate cultural knowledge and skills through social learning and generational training, outperforming single-lifetime training.
paper_ai_keywords: cultural accumulation reinforcement learning generational algorithms social learning in-context learning in-weights learning emergent cultural accumulation open-ended learning systems human culture