title: BigCodeBench: Benchmarking Code Generation with Diverse Function Calls
    and Complex Instructions
summary: Automated software engineering has been greatly empowered by the recent
  advances in Large Language Models (LLMs) for programming. While current
  benchmarks have shown that LLMs can perform various software engineering tasks
  like human developers, the majority of their evaluations are limited to short
  and self-contained algorithmic tasks. Solving challenging and practical
  programming tasks requires the capability of utilizing diverse function calls
  as tools to efficiently implement functionalities like data analysis and web
  development. In addition, using multiple tools to solve a task needs
  compositional reasoning by accurately understanding complex instructions.
  Fulfilling both of these characteristics can pose a great challenge for LLMs.
  To assess how well LLMs can solve challenging and practical programming tasks,
  we introduce Bench, a benchmark that challenges LLMs to invoke multiple
  function calls as tools from 139 libraries and 7 domains for 1,140 fine-grained
  programming tasks. To evaluate LLMs rigorously, each programming task
  encompasses 5.6 test cases with an average branch coverage of 99%. In addition,
  we propose a natural-language-oriented variant of Bench, Benchi, that
  automatically transforms the original docstrings into short instructions only
  with essential information. Our extensive evaluation of 60 LLMs shows that LLMs
  are not yet capable of following complex instructions to use function calls
  precisely, with scores up to 60%, significantly lower than the human
  performance of 97%. The results underscore the need for further advancements in
  this area.
paper_authors._id: 667a170251c8952e0769d84a
paper_authors.name: Terry Yue Zhuo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-25T07:44:50.867Z
paper_authors._id: 62b7fb545233925f253531c8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62b7fb545233925f253531c8/W50u2G1HK3EtUKHRU189V.jpeg
paper_authors.fullname: Terry Yue Zhuo
paper_authors.type: user
paper_authors.user: terryyz
paper_authors._id: 667a170251c8952e0769d84b
paper_authors.name: Minh Chien Vu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-25T20:03:56.684Z
paper_authors._id: 60535c9d10aba34e3b6a2ef7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1616075850230-noauth.jpeg
paper_authors.fullname: vumichien
paper_authors.type: user
paper_authors.user: vumichien
paper_authors._id: 667a170251c8952e0769d84c
paper_authors.name: Jenny Chim
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-25T07:45:08.786Z
paper_authors._id: 5fc8e6e9628d967e353416b3
paper_authors.avatarUrl: /avatars/d913dff9380e711aa4ecf368e3070f88.svg
paper_authors.fullname: Jenny Chim
paper_authors.type: user
paper_authors.user: j-chim
paper_authors._id: 667a170251c8952e0769d84d
paper_authors.name: Han Hu
paper_authors._id: 667a170251c8952e0769d84e
paper_authors.name: Wenhao Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-25T07:45:22.045Z
paper_authors._id: 5feab3a28a3201f8e554c969
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1660795228685-5feab3a28a3201f8e554c969.png
paper_authors.fullname: Wenhao Yu
paper_authors.type: user
paper_authors.user: wyu1
paper_authors._id: 667a170251c8952e0769d84f
paper_authors.name: Ratnadira Widyasari
paper_authors._id: 667a170251c8952e0769d850
paper_authors.name: Imam Nur Bani Yusuf
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-25T07:45:32.815Z
paper_authors._id: 61bd6bbb92dd56338582ebbd
paper_authors.avatarUrl: /avatars/3637ab04f9342cf15340a47c9645722a.svg
paper_authors.fullname: Imam Nur Bani Yusuf
paper_authors.type: user
paper_authors.user: imamnurby
paper_authors._id: 667a170251c8952e0769d851
paper_authors.name: Haolan Zhan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-25T07:45:39.609Z
paper_authors._id: 64fa937114636d417a87e2ff
paper_authors.avatarUrl: /avatars/3ca99b55bc920cf868657ec947e86a3f.svg
paper_authors.fullname: Haolan Zhan
paper_authors.type: user
paper_authors.user: zhanhaolan
paper_authors._id: 667a170251c8952e0769d852
paper_authors.name: Junda He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-25T07:45:45.375Z
paper_authors._id: 6208b9fbc51b8a11431762f0
paper_authors.avatarUrl: /avatars/a9983416adf916267cdc4b179da95336.svg
paper_authors.fullname: junda he
paper_authors.type: user
paper_authors.user: dennishe97
paper_authors._id: 667a170251c8952e0769d853
paper_authors.name: Indraneil Paul
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-25T07:45:51.159Z
paper_authors._id: 62054c0b522e40b4a18d8744
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62054c0b522e40b4a18d8744/e6bJyJCwEuOZI8BiKC0rx.jpeg
paper_authors.fullname: Indraneil Paul
paper_authors.type: user
paper_authors.user: iNeil77
paper_authors._id: 667a170251c8952e0769d854
paper_authors.name: Simon Brunner
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-25T07:45:56.727Z
paper_authors._id: 63fbd9841b4b1bd4e705bca1
paper_authors.avatarUrl: /avatars/f2ec3670bb984b9e17c2ebd4bb602e35.svg
paper_authors.fullname: Simon Brunner
paper_authors.type: user
paper_authors.user: SBRUNNE2
paper_authors._id: 667a170251c8952e0769d855
paper_authors.name: Chen Gong
paper_authors._id: 667a170251c8952e0769d856
paper_authors.name: Thong Hoang
paper_authors._id: 667a170251c8952e0769d857
paper_authors.name: Armel Randy Zebaze
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-25T07:46:11.542Z
paper_authors._id: 63952b31ee411d4c6fba891c
paper_authors.avatarUrl: /avatars/e901c3aaa16d3a06dff09896ce8a67a2.svg
paper_authors.fullname: Armel Randy Zebaze
paper_authors.type: user
paper_authors.user: ArmelRandy
paper_authors._id: 667a170251c8952e0769d858
paper_authors.name: Xiaoheng Hong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-25T07:46:18.546Z
paper_authors._id: 64c4fead9cff58f22c1dbd1e
paper_authors.avatarUrl: /avatars/95be1e289f8ba8b384a47ff65b506168.svg
paper_authors.fullname: XiaohengHong
paper_authors.type: user
paper_authors.user: hongxh
paper_authors._id: 667a170251c8952e0769d859
paper_authors.name: Wen-Ding Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-25T07:46:25.367Z
paper_authors._id: 62e221dfcb1f164f2cb8a66b
paper_authors.avatarUrl: /avatars/06f05622e232304d3f0b8c291f3263be.svg
paper_authors.fullname: Wen-Ding Li
paper_authors.type: user
paper_authors.user: xu3kev
paper_authors._id: 667a170251c8952e0769d85a
paper_authors.name: Jean Kaddour
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-25T07:46:37.344Z
paper_authors._id: 6304061c0547362a22a76a17
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1661339692442-6304061c0547362a22a76a17.jpeg
paper_authors.fullname: Jean Kaddour
paper_authors.type: user
paper_authors.user: JeanKaddour
paper_authors._id: 667a170251c8952e0769d85b
paper_authors.name: Ming Xu
paper_authors._id: 667a170251c8952e0769d85c
paper_authors.name: Zhihan Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-25T08:36:11.547Z
paper_authors._id: 63bf9695da08ed054400205e
paper_authors.avatarUrl: /avatars/b6fca49559a61cf66628088c60d26c10.svg
paper_authors.fullname: Zhihan Zhang
paper_authors.type: user
paper_authors.user: zhihz0535
paper_authors._id: 667a170251c8952e0769d85d
paper_authors.name: Prateek Yadav
paper_authors._id: 667a170251c8952e0769d85e
paper_authors.name: Naman Jain
paper_authors._id: 667a170251c8952e0769d85f
paper_authors.name: Alex Gu
paper_authors._id: 667a170251c8952e0769d860
paper_authors.name: Zhoujun Cheng
paper_authors._id: 667a170251c8952e0769d861
paper_authors.name: Jiawei Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-21T07:54:09.175Z
paper_authors._id: 644b584a9279988e0cbeb664
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/644b584a9279988e0cbeb664/fz4pn7HeXXfpvz23v8E8t.png
paper_authors.fullname: Jiawei Liu
paper_authors.type: user
paper_authors.user: ganler
paper_authors._id: 667a170251c8952e0769d862
paper_authors.name: Qian Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-25T20:03:58.673Z
paper_authors._id: 612ee6a7b960e78c6d2319d4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/612ee6a7b960e78c6d2319d4/2Hu9BaAyXbyh1vt0v1Qui.jpeg
paper_authors.fullname: Qian Liu
paper_authors.type: user
paper_authors.user: SivilTaram
paper_authors._id: 667a170251c8952e0769d863
paper_authors.name: Zijian Wang
paper_authors._id: 667a170251c8952e0769d864
paper_authors.name: David Lo
paper_authors._id: 667a170251c8952e0769d865
paper_authors.name: Binyuan Hui
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-23T11:12:34.605Z
paper_authors._id: 61e4c4ca1ab24785ac11ba69
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61e4c4ca1ab24785ac11ba69/1Q1zhhyGSJ9RJG9MzwxVv.jpeg
paper_authors.fullname: Binyuan Hui
paper_authors.type: user
paper_authors.user: huybery
paper_authors._id: 667a170251c8952e0769d866
paper_authors.name: Niklas Muennighoff
paper_authors._id: 667a170251c8952e0769d867
paper_authors.name: Daniel Fried
paper_authors._id: 667a170251c8952e0769d868
paper_authors.name: Xiaoning Du
paper_authors._id: 667a170251c8952e0769d869
paper_authors.name: Harm de Vries
paper_authors._id: 667a170251c8952e0769d86a
paper_authors.name: Leandro Von Werra
paper_title: BigCodeBench: Benchmarking Code Generation with Diverse Function Calls
    and Complex Instructions
paper_summary: Automated software engineering has been greatly empowered by the recent
  advances in Large Language Models (LLMs) for programming. While current
  benchmarks have shown that LLMs can perform various software engineering tasks
  like human developers, the majority of their evaluations are limited to short
  and self-contained algorithmic tasks. Solving challenging and practical
  programming tasks requires the capability of utilizing diverse function calls
  as tools to efficiently implement functionalities like data analysis and web
  development. In addition, using multiple tools to solve a task needs
  compositional reasoning by accurately understanding complex instructions.
  Fulfilling both of these characteristics can pose a great challenge for LLMs.
  To assess how well LLMs can solve challenging and practical programming tasks,
  we introduce Bench, a benchmark that challenges LLMs to invoke multiple
  function calls as tools from 139 libraries and 7 domains for 1,140 fine-grained
  programming tasks. To evaluate LLMs rigorously, each programming task
  encompasses 5.6 test cases with an average branch coverage of 99%. In addition,
  we propose a natural-language-oriented variant of Bench, Benchi, that
  automatically transforms the original docstrings into short instructions only
  with essential information. Our extensive evaluation of 60 LLMs shows that LLMs
  are not yet capable of following complex instructions to use function calls
  precisely, with scores up to 60%, significantly lower than the human
  performance of 97%. The results underscore the need for further advancements in
  this area.