title: Demystifying GPT Self-Repair for Code Generation
summary: Large Language Models (LLMs) have shown remarkable aptitude in code
  generation but still struggle on challenging programming tasks. Self-repair --
  in which the model debugs and fixes mistakes in its own code -- has recently
  become a popular way to boost performance in these settings. However, only very
  limited studies on how and when self-repair works effectively exist in the
  literature, and one might wonder to what extent a model is really capable of
  providing accurate feedback on why the code is wrong when that code was
  generated by the same model. In this paper, we analyze GPT-3.5 and GPT-4's
  ability to perform self-repair on APPS, a challenging dataset consisting of
  diverse coding challenges. To do so, we first establish a new evaluation
  strategy dubbed pass@t that measures the pass rate of the tasks against the
  total number of tokens sampled from the model, enabling a fair comparison to
  purely sampling-based approaches. With this evaluation strategy, we find that
  the effectiveness of self-repair is only seen in GPT-4. We also observe that
  self-repair is bottlenecked by the feedback stage; using GPT-4 to give feedback
  on the programs generated by GPT-3.5 and using expert human programmers to give
  feedback on the programs generated by GPT-4, we unlock significant performance
  gains.
paper_authors._id: 648fb5d34446e3b4c0d0992f
paper_authors.name: Theo X. Olausson
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-06-20T13:26:56.132Z
paper_authors._id: 6489e543b9e9258ba06d73de
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/Xjs0N338heFNkRkvHg9hL.jpeg
paper_authors.fullname: Theo X. Olausson
paper_authors.type: user
paper_authors.user: theoxo
paper_authors._id: 648fb5d34446e3b4c0d09930
paper_authors.name: Jeevana Priya Inala
paper_authors._id: 648fb5d34446e3b4c0d09931
paper_authors.name: Chenglong Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-19T07:17:28.060Z
paper_authors._id: 634f47584677a5891c101f10
paper_authors.avatarUrl: /avatars/9055ed8d27c80f21daef3ccab3b088cf.svg
paper_authors.fullname: Chenglong Wang
paper_authors.type: user
paper_authors.user: chenglong-ms
paper_authors._id: 648fb5d34446e3b4c0d09932
paper_authors.name: Jianfeng Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-19T07:18:00.244Z
paper_authors._id: 641904caf9d6f1d772ec7af7
paper_authors.avatarUrl: /avatars/4a63eac71eb30f70b1a0e9d4708f26c1.svg
paper_authors.fullname: Jianfeng Gao
paper_authors.type: user
paper_authors.user: wyngjf
paper_authors._id: 648fb5d34446e3b4c0d09933
paper_authors.name: Armando Solar-Lezama
paper_title: Demystifying GPT Self-Repair for Code Generation
paper_summary: Large Language Models (LLMs) have shown remarkable aptitude in code
  generation but still struggle on challenging programming tasks. Self-repair --
  in which the model debugs and fixes mistakes in its own code -- has recently
  become a popular way to boost performance in these settings. However, only very
  limited studies on how and when self-repair works effectively exist in the
  literature, and one might wonder to what extent a model is really capable of
  providing accurate feedback on why the code is wrong when that code was
  generated by the same model. In this paper, we analyze GPT-3.5 and GPT-4's
  ability to perform self-repair on APPS, a challenging dataset consisting of
  diverse coding challenges. To do so, we first establish a new evaluation
  strategy dubbed pass@t that measures the pass rate of the tasks against the
  total number of tokens sampled from the model, enabling a fair comparison to
  purely sampling-based approaches. With this evaluation strategy, we find that
  the effectiveness of self-repair is only seen in GPT-4. We also observe that
  self-repair is bottlenecked by the feedback stage; using GPT-4 to give feedback
  on the programs generated by GPT-3.5 and using expert human programmers to give
  feedback on the programs generated by GPT-4, we unlock significant performance
  gains.
paper_ai_keywords: Self-repair GPT-3.5 GPT-4 APPS pass@t feedback stage expert human programmers