Demystifying GPT Self-Repair for Code Generation
Large Language Models (LLMs) have shown remarkable aptitude in code
generation but still struggle on challenging programming tasks. Self-repair --
in which the model debugs and fixes mistakes in its own code -- has recently
become a popular way to boost performance in these settings. However, only very
limited studies on how and when self-repair works effectively exist in the
literature, and one might wonder to what extent a model is really capable of
providing accurate feedback on why the code is wrong when that code was
generated by the same model. In this paper, we analyze GPT-3.5 and GPT-4's
ability to perform self-repair on APPS, a challenging dataset consisting of
diverse coding challenges. To do so, we first establish a new evaluation
strategy dubbed pass@t that measures the pass rate of the tasks against the
total number of tokens sampled from the model, enabling a fair comparison to
purely sampling-based approaches. With this evaluation strategy, we find that
the effectiveness of self-repair is only seen in GPT-4. We also observe that
self-repair is bottlenecked by the feedback stage; using GPT-4 to give feedback
on the programs generated by GPT-3.5 and using expert human programmers to give
feedback on the programs generated by GPT-4, we unlock significant performance
gains.
648fb5d34446e3b4c0d0992f
Theo X. Olausson
extracted_confirmed
2023-06-20T13:26:56.132Z
6489e543b9e9258ba06d73de
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/Xjs0N338heFNkRkvHg9hL.jpeg
Theo X. Olausson
user
theoxo
648fb5d34446e3b4c0d09930
Jeevana Priya Inala
648fb5d34446e3b4c0d09931
Chenglong Wang
admin_assigned
2023-06-19T07:17:28.060Z
634f47584677a5891c101f10
/avatars/9055ed8d27c80f21daef3ccab3b088cf.svg
Chenglong Wang
user
chenglong-ms
648fb5d34446e3b4c0d09932
Jianfeng Gao
admin_assigned
2023-06-19T07:18:00.244Z
641904caf9d6f1d772ec7af7
/avatars/4a63eac71eb30f70b1a0e9d4708f26c1.svg
Jianfeng Gao
user
wyngjf
648fb5d34446e3b4c0d09933
Armando Solar-Lezama
Demystifying GPT Self-Repair for Code Generation
Large Language Models (LLMs) have shown remarkable aptitude in code
generation but still struggle on challenging programming tasks. Self-repair --
in which the model debugs and fixes mistakes in its own code -- has recently
become a popular way to boost performance in these settings. However, only very
limited studies on how and when self-repair works effectively exist in the
literature, and one might wonder to what extent a model is really capable of
providing accurate feedback on why the code is wrong when that code was
generated by the same model. In this paper, we analyze GPT-3.5 and GPT-4's
ability to perform self-repair on APPS, a challenging dataset consisting of
diverse coding challenges. To do so, we first establish a new evaluation
strategy dubbed pass@t that measures the pass rate of the tasks against the
total number of tokens sampled from the model, enabling a fair comparison to
purely sampling-based approaches. With this evaluation strategy, we find that
the effectiveness of self-repair is only seen in GPT-4. We also observe that
self-repair is bottlenecked by the feedback stage; using GPT-4 to give feedback
on the programs generated by GPT-3.5 and using expert human programmers to give
feedback on the programs generated by GPT-4, we unlock significant performance
gains.
GPT-4 demonstrates superior self-repair capabilities on APPS dataset compared to GPT-3.5, with performance boosted when feedback is provided by GPT-4 or human programmers.
Large Language Models LLMs code generation self-repair GPT-3.5 GPT-4 APPS dataset pass@t sampling-based approaches feedback stage