title: ML-Bench: Large Language Models Leverage Open-source Libraries for
    Machine Learning Tasks
summary: Large language models have shown promising performance in code generation
  benchmarks. However, a considerable divide exists between these benchmark
  achievements and their practical applicability, primarily attributed to
  real-world programming's reliance on pre-existing libraries. Instead of
  evaluating LLMs to code from scratch, this work aims to propose a new
  evaluation setup where LLMs use open-source libraries to finish machine
  learning tasks. Therefore, we propose ML-Bench, an expansive benchmark
  developed to assess the effectiveness of LLMs in leveraging existing functions
  in open-source libraries. Consisting of 10044 samples spanning 130 tasks over
  14 notable machine learning GitHub repositories. In this setting, given a
  specific machine learning task instruction and the accompanying README in a
  codebase, an LLM is tasked to generate code to accomplish the task. This
  necessitates the comprehension of long and language-code interleaved documents,
  as well as the understanding of complex cross-file code structures, introducing
  new challenges. Notably, while GPT-4 exhibits remarkable improvement over other
  LLMs, it manages to accomplish only 39.73\% of the tasks, leaving a huge space
  for improvement. We address these challenges by proposing ML-Agent, designed to
  effectively navigate the codebase, locate documentation, retrieve code, and
  generate executable code. Empirical results demonstrate that ML-Agent, built
  upon GPT-4, results in further improvements. Code, data, and models are
  available at https://ml-bench.github.io/.
paper_authors._id: 6556dcf347c93d37a885c33e
paper_authors.name: Yuliang Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-17T16:28:44.735Z
paper_authors._id: 64b74e149f5987572c9bf8d8
paper_authors.avatarUrl: /avatars/5429714d53a1bfdf6a6c7f077b3b5cf6.svg
paper_authors.fullname: Yuliang Liu
paper_authors.type: user
paper_authors.user: YuliangLiu
paper_authors._id: 6556dcf347c93d37a885c33f
paper_authors.name: Xiangru Tang
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-11-20T03:25:55.732Z
paper_authors._id: 63357c608adfa81faf2ac180
paper_authors.avatarUrl: /avatars/ae0314c644f882251baf59b9134fd36f.svg
paper_authors.fullname: Xiangru Tang
paper_authors.type: user
paper_authors.user: RTT1
paper_authors._id: 6556dcf347c93d37a885c340
paper_authors.name: Zefan Cai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-17T16:28:51.397Z
paper_authors._id: 64b15284372d4340772a3dca
paper_authors.avatarUrl: /avatars/417d5f1bc1bcb5e4d5de6169673c2cf7.svg
paper_authors.fullname: Zefan Cai
paper_authors.type: user
paper_authors.user: ZefanCai
paper_authors._id: 6556dcf347c93d37a885c341
paper_authors.name: Junjie Lu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-24T13:05:43.940Z
paper_authors._id: 660e1bb6e089e44d2906a7e8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/660e1bb6e089e44d2906a7e8/MJU8nHQ0xgiPzO8J6L8EC.jpeg
paper_authors.fullname: Junjie Lu
paper_authors.type: user
paper_authors.user: Lux0926
paper_authors._id: 6556dcf347c93d37a885c342
paper_authors.name: Yichi Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-17T16:30:15.504Z
paper_authors._id: 6369b1d456d1f93498130a8a
paper_authors.avatarUrl: /avatars/8ec228aa6f171715652511f948765db9.svg
paper_authors.fullname: Yichi Zhang
paper_authors.type: user
paper_authors.user: 594zyc
paper_authors._id: 6556dcf347c93d37a885c343
paper_authors.name: Yanjun Shao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-17T16:30:30.870Z
paper_authors._id: 62970df979f193515da13dc0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62970df979f193515da13dc0/A-mgKIcgTXRJ54GCHswTq.jpeg
paper_authors.fullname: Yanjun Shao
paper_authors.type: user
paper_authors.user: super-dainiu
paper_authors._id: 6556dcf347c93d37a885c344
paper_authors.name: Zexuan Deng
paper_authors._id: 6556dcf347c93d37a885c345
paper_authors.name: Helan Hu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-17T16:32:05.953Z
paper_authors._id: 6491aec41e161baf7894b54b
paper_authors.avatarUrl: /avatars/0af10e24669125e4329c003fc27b603e.svg
paper_authors.fullname: huhelan
paper_authors.type: user
paper_authors.user: hhhllan
paper_authors._id: 6556dcf347c93d37a885c346
paper_authors.name: Zengxian Yang
paper_authors._id: 6556dcf347c93d37a885c347
paper_authors.name: Kaikai An
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-23T07:37:40.004Z
paper_authors._id: 66c89152d33e34fbc29497d7
paper_authors.avatarUrl: /avatars/bbddabf6532393951c4759e5915a065b.svg
paper_authors.fullname: KaikaiAn
paper_authors.type: user
paper_authors.user: kkk-an
paper_authors._id: 6556dcf347c93d37a885c348
paper_authors.name: Ruijun Huang
paper_authors._id: 6556dcf347c93d37a885c349
paper_authors.name: Shuzheng Si
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-11T07:47:09.953Z
paper_authors._id: 637c99bbfe115289cfedfb44
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/637c99bbfe115289cfedfb44/344NN9KKF_XXTlVYaGaMW.png
paper_authors.fullname: ssz
paper_authors.type: user
paper_authors.user: ssz1111
paper_authors._id: 6556dcf347c93d37a885c34a
paper_authors.name: Sheng Chen
paper_authors._id: 6556dcf347c93d37a885c34b
paper_authors.name: Haozhe Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-22T08:43:47.916Z
paper_authors._id: 63ff09a02098b9ad105a09f6
paper_authors.avatarUrl: /avatars/4409ca5d320050cf4c3df05962c7ff58.svg
paper_authors.fullname: Hans Zhao
paper_authors.type: user
paper_authors.user: BleachNick
paper_authors._id: 6556dcf347c93d37a885c34c
paper_authors.name: Zhengliang Li
paper_authors._id: 6556dcf347c93d37a885c34d
paper_authors.name: Liang Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-12T12:06:38.496Z
paper_authors._id: 61b0a4ce1b3d95b3d1ed9251
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/Wwjr26vdudX5KYVTb8Q0a.png
paper_authors.fullname: Liang Chen
paper_authors.type: user
paper_authors.user: leonardPKU
paper_authors._id: 6556dcf347c93d37a885c34e
paper_authors.name: Yiming Zong
paper_authors._id: 6556dcf347c93d37a885c34f
paper_authors.name: Yan Wang
paper_authors._id: 6556dcf347c93d37a885c350
paper_authors.name: Tianyu Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-17T16:34:35.651Z
paper_authors._id: 62fc97c2ee999004b5a8fec2
paper_authors.avatarUrl: /avatars/0a7805c014c26b36e607155c30620142.svg
paper_authors.fullname: Tianyu Liu
paper_authors.type: user
paper_authors.user: tliu
paper_authors._id: 6556dcf347c93d37a885c351
paper_authors.name: Zhiwei Jiang
paper_authors._id: 6556dcf347c93d37a885c352
paper_authors.name: Baobao Chang
paper_authors._id: 6556dcf347c93d37a885c353
paper_authors.name: Yujia Qin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-17T16:35:03.572Z
paper_authors._id: 643f37cce9d063936912048b
paper_authors.avatarUrl: /avatars/25822ea5676a79b2e1ddf08d5fc2226c.svg
paper_authors.fullname: Yujia Qin
paper_authors.type: user
paper_authors.user: YujiaHi
paper_authors._id: 6556dcf347c93d37a885c354
paper_authors.name: Wangchunshu Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-17T16:35:13.290Z
paper_authors._id: 628c8598ef14f971b698107f
paper_authors.avatarUrl: /avatars/3a4ad87e6b5f9e836a1160d869df1447.svg
paper_authors.fullname: Zhou
paper_authors.type: user
paper_authors.user: Wangchunshu
paper_authors._id: 6556dcf347c93d37a885c355
paper_authors.name: Yilun Zhao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-17T16:35:20.914Z
paper_authors._id: 62f662bcc58915315c4eccea
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62f662bcc58915315c4eccea/zOAQLONfMP88zr70sxHK-.jpeg
paper_authors.fullname: Yilun
paper_authors.type: user
paper_authors.user: yilunzhao
paper_authors._id: 6556dcf347c93d37a885c356
paper_authors.name: Arman Cohan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-17T16:35:31.956Z
paper_authors._id: 5f5ba21188f57f65f951f255
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1599840760465-noauth.png
paper_authors.fullname: Arman Cohan
paper_authors.type: user
paper_authors.user: armanc
paper_authors._id: 6556dcf347c93d37a885c357
paper_authors.name: Mark Gerstein
paper_title: ML-Bench: Large Language Models Leverage Open-source Libraries for
    Machine Learning Tasks
paper_summary: Large language models have shown promising performance in code generation
  benchmarks. However, a considerable divide exists between these benchmark
  achievements and their practical applicability, primarily attributed to
  real-world programming's reliance on pre-existing libraries. Instead of
  evaluating LLMs to code from scratch, this work aims to propose a new
  evaluation setup where LLMs use open-source libraries to finish machine
  learning tasks. Therefore, we propose ML-Bench, an expansive benchmark
  developed to assess the effectiveness of LLMs in leveraging existing functions
  in open-source libraries. Consisting of 10044 samples spanning 130 tasks over
  14 notable machine learning GitHub repositories. In this setting, given a
  specific machine learning task instruction and the accompanying README in a
  codebase, an LLM is tasked to generate code to accomplish the task. This
  necessitates the comprehension of long and language-code interleaved documents,
  as well as the understanding of complex cross-file code structures, introducing
  new challenges. Notably, while GPT-4 exhibits remarkable improvement over other
  LLMs, it manages to accomplish only 39.73\% of the tasks, leaving a huge space
  for improvement. We address these challenges by proposing ML-Agent, designed to
  effectively navigate the codebase, locate documentation, retrieve code, and
  generate executable code. Empirical results demonstrate that ML-Agent, built
  upon GPT-4, results in further improvements. Code, data, and models are
  available at https://ml-bench.github.io/.
paper_ai_keywords: ML-Bench ML-Agent large language models code generation open-source libraries machine learning tasks README long and language-code interleaved documents complex cross-file code structures executable code GPT-4