title: CodeIt: Self-Improving Language Models with Prioritized Hindsight Replay
summary: Large language models are increasingly solving tasks that are commonly
  believed to require human-level reasoning ability. However, these models still
  perform very poorly on benchmarks of general intelligence such as the
  Abstraction and Reasoning Corpus (ARC). In this paper, we approach ARC as a
  programming-by-examples problem, and introduce a novel and scalable method for
  language model self-improvement called Code Iteration (CodeIt). Our method
  iterates between 1) program sampling and hindsight relabeling, and 2) learning
  from prioritized experience replay. By relabeling the goal of an episode (i.e.,
  the target program output given input) to the realized output produced by the
  sampled program, our method effectively deals with the extreme sparsity of
  rewards in program synthesis. Applying CodeIt to the ARC dataset, we
  demonstrate that prioritized hindsight replay, along with pre-training and
  data-augmentation, leads to successful inter-task generalization. CodeIt is the
  first neuro-symbolic approach that scales to the full ARC evaluation dataset.
  Our method solves 15% of ARC evaluation tasks, achieving state-of-the-art
  performance and outperforming existing neural and symbolic baselines.
paper_authors._id: 65c44ced9f6c4689fbac4ecf
paper_authors.name: Natasha Butt
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-31T09:55:49.570Z
paper_authors._id: 67235358177d9eae488ff01b
paper_authors.avatarUrl: /avatars/b94e228ceae5155898af4a6d1bb06ed3.svg
paper_authors.fullname: Natasha Butt
paper_authors.type: user
paper_authors.user: NatashaEB
paper_authors._id: 65c44ced9f6c4689fbac4ed0
paper_authors.name: Blazej Manczak
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-08T13:01:42.918Z
paper_authors._id: 6054ba9a3efc404ddc7c0c20
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1677092793838-6054ba9a3efc404ddc7c0c20.jpeg
paper_authors.fullname: Blazej
paper_authors.type: user
paper_authors.user: Blazej
paper_authors._id: 65c44ced9f6c4689fbac4ed1
paper_authors.name: Auke Wiggers
paper_authors._id: 65c44ced9f6c4689fbac4ed2
paper_authors.name: Corrado Rainone
paper_authors._id: 65c44ced9f6c4689fbac4ed3
paper_authors.name: David Zhang
paper_authors._id: 65c44ced9f6c4689fbac4ed4
paper_authors.name: Michaël Defferrard
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-25T20:04:53.550Z
paper_authors._id: 666aefbce7c15edd93433846
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/666aefbce7c15edd93433846/ET_a5cdmDizqJ_EEEzYOY.jpeg
paper_authors.fullname: Michaël Defferrard
paper_authors.type: user
paper_authors.user: mdeff
paper_authors._id: 65c44ced9f6c4689fbac4ed5
paper_authors.name: Taco Cohen
paper_title: CodeIt: Self-Improving Language Models with Prioritized Hindsight Replay
paper_summary: Large language models are increasingly solving tasks that are commonly
  believed to require human-level reasoning ability. However, these models still
  perform very poorly on benchmarks of general intelligence such as the
  Abstraction and Reasoning Corpus (ARC). In this paper, we approach ARC as a
  programming-by-examples problem, and introduce a novel and scalable method for
  language model self-improvement called Code Iteration (CodeIt). Our method
  iterates between 1) program sampling and hindsight relabeling, and 2) learning
  from prioritized experience replay. By relabeling the goal of an episode (i.e.,
  the target program output given input) to the realized output produced by the
  sampled program, our method effectively deals with the extreme sparsity of
  rewards in program synthesis. Applying CodeIt to the ARC dataset, we
  demonstrate that prioritized hindsight replay, along with pre-training and
  data-augmentation, leads to successful inter-task generalization. CodeIt is the
  first neuro-symbolic approach that scales to the full ARC evaluation dataset.
  Our method solves 15% of ARC evaluation tasks, achieving state-of-the-art
  performance and outperforming existing neural and symbolic baselines.
paper_ai_summary: A Code Iteration method for language models achieves state-of-the-art performance on the Abstraction and Reasoning Corpus by combining program sampling, hindsight relabeling, and prioritized experience replay.
paper_ai_keywords: programming-by-examples Code Iteration program sampling hindsight relabeling learning from prioritized experience replay extreme sparsity of rewards program synthesis prioritized hindsight replay pre-training data-augmentation inter-task generalization neuro-symbolic approach Abstraction and Reasoning Corpus