title: Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving
summary: The task of issue resolving is to modify a codebase to generate a patch that
  addresses a given issue. However, existing benchmarks, such as SWE-bench, focus
  almost exclusively on Python, making them insufficient for evaluating Large
  Language Models (LLMs) across diverse software ecosystems. To address this, we
  introduce a multilingual issue-resolving benchmark, called Multi-SWE-bench,
  covering Java, TypeScript, JavaScript, Go, Rust, C, and C++. It includes a
  total of 1,632 high-quality instances, which were carefully annotated from
  2,456 candidates by 68 expert annotators, ensuring that the benchmark can
  provide an accurate and reliable evaluation. Based on Multi-SWE-bench, we
  evaluate a series of state-of-the-art models using three representative methods
  (Agentless, SWE-agent, and OpenHands) and present a comprehensive analysis with
  key empirical insights. In addition, we launch a Multi-SWE-RL open-source
  community, aimed at building large-scale reinforcement learning (RL) training
  datasets for issue-resolving tasks. As an initial contribution, we release a
  set of 4,723 well-structured instances spanning seven programming languages,
  laying a solid foundation for RL research in this domain. More importantly, we
  open-source our entire data production pipeline, along with detailed tutorials,
  encouraging the open-source community to continuously contribute and expand the
  dataset. We envision our Multi-SWE-bench and the ever-growing Multi-SWE-RL
  community as catalysts for advancing RL toward its full potential, bringing us
  one step closer to the dawn of AGI.
paper_authors._id: 67ef4d92c1e251f239495a13
paper_authors.name: Daoguang Zan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-08T06:56:29.085Z
paper_authors._id: 61527edf8b55dbdae72874fa
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61527edf8b55dbdae72874fa/ZGWSBf_KSrDof6WyMoDMU.jpeg
paper_authors.fullname: Daoguang Zan
paper_authors.type: user
paper_authors.user: Daoguang
paper_authors._id: 67ef4d92c1e251f239495a14
paper_authors.name: Zhirong Huang
paper_authors._id: 67ef4d92c1e251f239495a15
paper_authors.name: Wei Liu
paper_authors._id: 67ef4d92c1e251f239495a16
paper_authors.name: Hanwu Chen
paper_authors._id: 67ef4d92c1e251f239495a17
paper_authors.name: Linhao Zhang
paper_authors._id: 67ef4d92c1e251f239495a18
paper_authors.name: Shulin Xin
paper_authors._id: 67ef4d92c1e251f239495a19
paper_authors.name: Lu Chen
paper_authors._id: 67ef4d92c1e251f239495a1a
paper_authors.name: Qi Liu
paper_authors._id: 67ef4d92c1e251f239495a1b
paper_authors.name: Xiaojian Zhong
paper_authors._id: 67ef4d92c1e251f239495a1c
paper_authors.name: Aoyan Li
paper_authors._id: 67ef4d92c1e251f239495a1d
paper_authors.name: Siyao Liu
paper_authors._id: 67ef4d92c1e251f239495a1e
paper_authors.name: Yongsheng Xiao
paper_authors._id: 67ef4d92c1e251f239495a1f
paper_authors.name: Liangqiang Chen
paper_authors._id: 67ef4d92c1e251f239495a20
paper_authors.name: Yuyu Zhang
paper_authors._id: 67ef4d92c1e251f239495a21
paper_authors.name: Jing Su
paper_authors._id: 67ef4d92c1e251f239495a22
paper_authors.name: Tianyu Liu
paper_authors._id: 67ef4d92c1e251f239495a23
paper_authors.name: Rui Long
paper_authors._id: 67ef4d92c1e251f239495a24
paper_authors.name: Kai Shen
paper_authors._id: 67ef4d92c1e251f239495a25
paper_authors.name: Liang Xiang
paper_title: Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving
paper_summary: The task of issue resolving is to modify a codebase to generate a patch that
  addresses a given issue. However, existing benchmarks, such as SWE-bench, focus
  almost exclusively on Python, making them insufficient for evaluating Large
  Language Models (LLMs) across diverse software ecosystems. To address this, we
  introduce a multilingual issue-resolving benchmark, called Multi-SWE-bench,
  covering Java, TypeScript, JavaScript, Go, Rust, C, and C++. It includes a
  total of 1,632 high-quality instances, which were carefully annotated from
  2,456 candidates by 68 expert annotators, ensuring that the benchmark can
  provide an accurate and reliable evaluation. Based on Multi-SWE-bench, we
  evaluate a series of state-of-the-art models using three representative methods
  (Agentless, SWE-agent, and OpenHands) and present a comprehensive analysis with
  key empirical insights. In addition, we launch a Multi-SWE-RL open-source
  community, aimed at building large-scale reinforcement learning (RL) training
  datasets for issue-resolving tasks. As an initial contribution, we release a
  set of 4,723 well-structured instances spanning seven programming languages,
  laying a solid foundation for RL research in this domain. More importantly, we
  open-source our entire data production pipeline, along with detailed tutorials,
  encouraging the open-source community to continuously contribute and expand the
  dataset. We envision our Multi-SWE-bench and the ever-growing Multi-SWE-RL
  community as catalysts for advancing RL toward its full potential, bringing us
  one step closer to the dawn of AGI.
paper_ai_summary: A multilingual benchmark, Multi-SWE-bench, is introduced to evaluate Large Language Models across various programming languages and is complemented by a RL dataset, Multi-SWE-RL, to advance issue-resolving tasks.
paper_ai_keywords: Large Language Models (LLMs) multilingual issue-resolving benchmark Multi-SWE-bench Java TypeScript JavaScript Go Rust C C++ high-quality instances expert annotators state-of-the-art models Agentless SWE-agent OpenHands reinforcement learning (RL) Multi-SWE-RL open-source community data production pipeline detail tutorials AGI