What Makes Large Language Models Reason in (Multi-Turn) Code Generation?
Kunhao Zheng Juliette Decugis Jonas Gehring Taco Cohen benjamin negrevergne Gabriel Synnaeve
Prompting techniques such as chain-of-thought have established themselves as a popular vehicle for improving the outputs of large language models (LLMs). For code generation, however, their exact mechanics and efficacy are under-explored using unified metrics and benchmarks. We thus investigate the effects of a wide range of prompting strategies with a focus on automatic re-prompting over multiple turns and computational requirements. After systematically decomposing reasoning, instruction, and execution feedback prompts, we conduct an extensive grid search on the competitive programming benchmarks CodeContests and TACO for multiple LLM families and sizes (Llama 3.0 and 3.1, 8B, 70B, 405B, and GPT-4o). Our study reveals strategies that consistently improve performance across all models with small and large sampling budgets. We then show how finetuning with such an optimal configuration allows models to internalize the induced reasoning process and obtain improvements in performance and scalability for multi-turn code generation.
Large language Models, Multi-turn Code Generation, Chain-of-Thought
We break down the main components of recent CoT and self-repair techniques for multi-turn code generation identifying the most promising ones across benchmarks and models.