On Evaluating the Efficiency of Source Code Generated by LLMs
<|reference_start|>On Evaluating the Efficiency of Source Code Generated by LLMs: Recent years have seen the remarkable capabilities of large language models (LLMs) for code generation. Different from existing work that evaluate the correctness of the code generated by LLMs, we propose to further evaluate its efficiency. More efficient code can lead to higher performance and execution efficiency of programs and software completed by LLM-assisted programming. First, we evaluate the efficiency of the code generated by LLMs on two benchmarks, HumanEval and MBPP. Then, we choose a set of programming problems from the online judge platform LeetCode to conduct a more difficult evaluation. Finally, we explore several prompts that would enable LLMs to generate more efficient code.<|reference_end|>
@article{niu2024on,
    title={On Evaluating the Efficiency of Source Code Generated by LLMs},
    author={Changan Niu, Ting Zhang, Chuanyi Li, Bin Luo, Vincent Ng},
    journal={arXiv preprint arXiv:2404.06041},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.06041},
    primaryClass={cs.SE}
}