DOCE: Finding the Sweet Spot for Execution-Based Code Generation
<|reference_start|>DOCE: Finding the Sweet Spot for Execution-Based Code Generation: Recently, a diverse set of decoding and reranking procedures have been shown effective for LLM-based code generation. However, a comprehensive framework that links and experimentally compares these methods is missing. We address this by proposing Decoding Objectives for Code Execution, a comprehensive framework that includes candidate generation, $n$-best reranking, minimum Bayes risk (MBR) decoding, and self-debugging as the core components. We then study the contributions of these components through execution-based evaluation metrics. Our findings highlight the importance of execution-based methods and the difference gap between execution-based and execution-free methods. Furthermore, we assess the impact of filtering based on trial unit tests, a simple and effective strategy that has been often overlooked in prior works. We also propose self-debugging on multiple candidates, obtaining state-of-the-art performance on reranking for code generation. We expect our framework to provide a solid guideline for future research on code generation.<|reference_end|>
@article{li2024doce:,
    title={DOCE: Finding the Sweet Spot for Execution-Based Code Generation},
    author={Haau-Sing Li, Patrick Fernandes, Iryna Gurevych, Andr'e F.T. Martins},
    journal={arXiv preprint arXiv:2408.13745},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.13745},
    primaryClass={cs.CL cs.AI cs.PL}
}