BigCodeBench: Benchmarking Code Generation with Diverse Function Calls
  and Complex Instructions
Automated software engineering has been greatly empowered by the recent
advances in Large Language Models (LLMs) for programming. While current
benchmarks have shown that LLMs can perform various software engineering tasks
like human developers, the majority of their evaluations are limited to short
and self-contained algorithmic tasks. Solving challenging and practical
programming tasks requires the capability of utilizing diverse function calls
as tools to efficiently implement functionalities like data analysis and web
development. In addition, using multiple tools to solve a task needs
compositional reasoning by accurately understanding complex instructions.
Fulfilling both of these characteristics can pose a great challenge for LLMs.
To assess how well LLMs can solve challenging and practical programming tasks,
we introduce Bench, a benchmark that challenges LLMs to invoke multiple
function calls as tools from 139 libraries and 7 domains for 1,140 fine-grained
programming tasks. To evaluate LLMs rigorously, each programming task
encompasses 5.6 test cases with an average branch coverage of 99%. In addition,
we propose a natural-language-oriented variant of Bench, Benchi, that
automatically transforms the original docstrings into short instructions only
with essential information. Our extensive evaluation of 60 LLMs shows that LLMs
are not yet capable of following complex instructions to use function calls
precisely, with scores up to 60%, significantly lower than the human
performance of 97%. The results underscore the need for further advancements in
this area.
667a170251c8952e0769d84a
Terry Yue Zhuo
admin_assigned
2024-06-25T07:44:50.867Z
62b7fb545233925f253531c8
https://cdn-avatars.huggingface.co/v1/production/uploads/62b7fb545233925f253531c8/W50u2G1HK3EtUKHRU189V.jpeg
Terry Yue Zhuo
user
terryyz
667a170251c8952e0769d84b
Minh Chien Vu
claimed_verified
2024-06-25T20:03:56.684Z
60535c9d10aba34e3b6a2ef7
https://cdn-avatars.huggingface.co/v1/production/uploads/1616075850230-noauth.jpeg
vumichien
user
vumichien
667a170251c8952e0769d84c
Jenny Chim
admin_assigned
2024-06-25T07:45:08.786Z
5fc8e6e9628d967e353416b3
/avatars/d913dff9380e711aa4ecf368e3070f88.svg
Jenny Chim
user
j-chim
667a170251c8952e0769d84d
Han Hu
667a170251c8952e0769d84e
Wenhao Yu
admin_assigned
2024-06-25T07:45:22.045Z
5feab3a28a3201f8e554c969
https://cdn-avatars.huggingface.co/v1/production/uploads/1660795228685-5feab3a28a3201f8e554c969.png
Wenhao Yu
user
wyu1
667a170251c8952e0769d84f
Ratnadira Widyasari
667a170251c8952e0769d850
Imam Nur Bani Yusuf
admin_assigned
2024-06-25T07:45:32.815Z
61bd6bbb92dd56338582ebbd
/avatars/3637ab04f9342cf15340a47c9645722a.svg
Imam Nur Bani Yusuf
user
imamnurby
667a170251c8952e0769d851
Haolan Zhan
admin_assigned
2024-06-25T07:45:39.609Z
64fa937114636d417a87e2ff
/avatars/3ca99b55bc920cf868657ec947e86a3f.svg
Haolan Zhan
user
zhanhaolan
667a170251c8952e0769d852
Junda He
admin_assigned
2024-06-25T07:45:45.375Z
6208b9fbc51b8a11431762f0
/avatars/a9983416adf916267cdc4b179da95336.svg
junda he
user
dennishe97
667a170251c8952e0769d853
Indraneil Paul
admin_assigned
2024-06-25T07:45:51.159Z
62054c0b522e40b4a18d8744
https://cdn-avatars.huggingface.co/v1/production/uploads/62054c0b522e40b4a18d8744/e6bJyJCwEuOZI8BiKC0rx.jpeg
Indraneil Paul
user
iNeil77
667a170251c8952e0769d854
Simon Brunner
admin_assigned
2024-06-25T07:45:56.727Z
63fbd9841b4b1bd4e705bca1
/avatars/f2ec3670bb984b9e17c2ebd4bb602e35.svg
Simon Brunner
user
SBRUNNE2
667a170251c8952e0769d855
Chen Gong
667a170251c8952e0769d856
Thong Hoang
667a170251c8952e0769d857
Armel Randy Zebaze
admin_assigned
2024-06-25T07:46:11.542Z
63952b31ee411d4c6fba891c
/avatars/e901c3aaa16d3a06dff09896ce8a67a2.svg
Armel Randy Zebaze
user
ArmelRandy
667a170251c8952e0769d858
Xiaoheng Hong
admin_assigned
2024-06-25T07:46:18.546Z
64c4fead9cff58f22c1dbd1e
/avatars/95be1e289f8ba8b384a47ff65b506168.svg
XiaohengHong
user
hongxh
667a170251c8952e0769d859
Wen-Ding Li
admin_assigned
2024-06-25T07:46:25.367Z
62e221dfcb1f164f2cb8a66b
/avatars/06f05622e232304d3f0b8c291f3263be.svg
Wen-Ding Li
user
xu3kev
667a170251c8952e0769d85a
Jean Kaddour
admin_assigned
2024-06-25T07:46:37.344Z
6304061c0547362a22a76a17
https://cdn-avatars.huggingface.co/v1/production/uploads/1661339692442-6304061c0547362a22a76a17.jpeg
Jean Kaddour
user
JeanKaddour
667a170251c8952e0769d85b
Ming Xu
667a170251c8952e0769d85c
Zhihan Zhang
claimed_verified
2024-06-25T08:36:11.547Z
63bf9695da08ed054400205e
/avatars/5223b212431337c6186db96ac70b7ae7.svg
Zhihan Zhang
user
zhihz0535
667a170251c8952e0769d85d
Prateek Yadav
667a170251c8952e0769d85e
Naman Jain
667a170251c8952e0769d85f
Alex Gu
667a170251c8952e0769d860
Zhoujun Cheng
667a170251c8952e0769d861
Jiawei Liu
claimed_verified
2024-10-21T07:54:09.175Z
644b584a9279988e0cbeb664
https://cdn-avatars.huggingface.co/v1/production/uploads/644b584a9279988e0cbeb664/fhWCI_Q26tTruhdFkjejw.jpeg
Jiawei Liu
user
ganler
667a170251c8952e0769d862
Qian Liu
claimed_verified
2024-06-25T20:03:58.673Z
612ee6a7b960e78c6d2319d4
https://cdn-avatars.huggingface.co/v1/production/uploads/612ee6a7b960e78c6d2319d4/2Hu9BaAyXbyh1vt0v1Qui.jpeg
Qian Liu
user
SivilTaram
667a170251c8952e0769d863
Zijian Wang
667a170251c8952e0769d864
David Lo
667a170251c8952e0769d865
Binyuan Hui
claimed_verified
2024-12-23T11:12:34.605Z
61e4c4ca1ab24785ac11ba69
https://cdn-avatars.huggingface.co/v1/production/uploads/61e4c4ca1ab24785ac11ba69/1Q1zhhyGSJ9RJG9MzwxVv.jpeg
Binyuan Hui
user
huybery
667a170251c8952e0769d866
Niklas Muennighoff
667a170251c8952e0769d867
Daniel Fried
667a170251c8952e0769d868
Xiaoning Du
667a170251c8952e0769d869
Harm de Vries
667a170251c8952e0769d86a
Leandro Von Werra
BigCodeBench: Benchmarking Code Generation with Diverse Function Calls
  and Complex Instructions
Automated software engineering has been greatly empowered by the recent
advances in Large Language Models (LLMs) for programming. While current
benchmarks have shown that LLMs can perform various software engineering tasks
like human developers, the majority of their evaluations are limited to short
and self-contained algorithmic tasks. Solving challenging and practical
programming tasks requires the capability of utilizing diverse function calls
as tools to efficiently implement functionalities like data analysis and web
development. In addition, using multiple tools to solve a task needs
compositional reasoning by accurately understanding complex instructions.
Fulfilling both of these characteristics can pose a great challenge for LLMs.
To assess how well LLMs can solve challenging and practical programming tasks,
we introduce Bench, a benchmark that challenges LLMs to invoke multiple
function calls as tools from 139 libraries and 7 domains for 1,140 fine-grained
programming tasks. To evaluate LLMs rigorously, each programming task
encompasses 5.6 test cases with an average branch coverage of 99%. In addition,
we propose a natural-language-oriented variant of Bench, Benchi, that
automatically transforms the original docstrings into short instructions only
with essential information. Our extensive evaluation of 60 LLMs shows that LLMs
are not yet capable of following complex instructions to use function calls
precisely, with scores up to 60%, significantly lower than the human
performance of 97%. The results underscore the need for further advancements in
this area.
Benchmark Bench assesses LLMs' capability to solve practical programming tasks by invoking function calls from various libraries, revealing significant performance gaps compared to human developers.
Large Language Models (LLMs) Bench Benchi function calls compositional reasoning test cases branch coverage natural-language-oriented variant programming tasks