title: Is Programming by Example solved by LLMs?
summary: Programming-by-Examples (PBE) aims to generate an algorithm from input-output
  examples. Such systems are practically and theoretically important: from an
  end-user perspective, they are deployed to millions of people, and from an AI
  perspective, PBE corresponds to a very general form of few-shot inductive
  inference. Given the success of Large Language Models (LLMs) in code-generation
  tasks, we investigate here the extent to which LLMs can be said to have
  `solved' PBE. We experiment on classic domains such as lists and strings, and
  an uncommon graphics programming domain not well represented in typical
  pretraining data. We find that pretrained models are not effective at PBE, but
  that they can be fine-tuned for much higher performance, provided the test
  problems are in-distribution. We analyze empirically what causes these models
  to succeed and fail, and take steps toward understanding how to achieve better
  out-of-distribution generalization. Collectively these results suggest that
  LLMs make strong progress toward solving the typical suite of PBE tasks,
  potentially increasing the flexibility and applicability of PBE systems, while
  also identifying ways in which LLMs still fall short.
paper_authors._id: 666a633a3536e9a956be85aa
paper_authors.name: Wen-Ding Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-28T08:03:13.875Z
paper_authors._id: 62e221dfcb1f164f2cb8a66b
paper_authors.avatarUrl: /avatars/06f05622e232304d3f0b8c291f3263be.svg
paper_authors.fullname: Wen-Ding Li
paper_authors.type: user
paper_authors.user: xu3kev
paper_authors._id: 666a633a3536e9a956be85ab
paper_authors.name: Kevin Ellis
paper_title: Is Programming by Example solved by LLMs?
paper_summary: Programming-by-Examples (PBE) aims to generate an algorithm from input-output
  examples. Such systems are practically and theoretically important: from an
  end-user perspective, they are deployed to millions of people, and from an AI
  perspective, PBE corresponds to a very general form of few-shot inductive
  inference. Given the success of Large Language Models (LLMs) in code-generation
  tasks, we investigate here the extent to which LLMs can be said to have
  `solved' PBE. We experiment on classic domains such as lists and strings, and
  an uncommon graphics programming domain not well represented in typical
  pretraining data. We find that pretrained models are not effective at PBE, but
  that they can be fine-tuned for much higher performance, provided the test
  problems are in-distribution. We analyze empirically what causes these models
  to succeed and fail, and take steps toward understanding how to achieve better
  out-of-distribution generalization. Collectively these results suggest that
  LLMs make strong progress toward solving the typical suite of PBE tasks,
  potentially increasing the flexibility and applicability of PBE systems, while
  also identifying ways in which LLMs still fall short.
paper_ai_summary: Large Language Models improve Programming-by-Examples performance through fine-tuning but still struggle with out-of-distribution tasks.
paper_ai_keywords: Programming-by-Examples Large Language Models few-shot inductive inference code-generation fine-tuning in-distribution out-of-distribution generalization