title: SWE-bench-java: A GitHub Issue Resolving Benchmark for Java
summary: GitHub issue resolving is a critical task in software engineering, recently
  gaining significant attention in both industry and academia. Within this task,
  SWE-bench has been released to evaluate issue resolving capabilities of large
  language models (LLMs), but has so far only focused on Python version. However,
  supporting more programming languages is also important, as there is a strong
  demand in industry. As a first step toward multilingual support, we have
  developed a Java version of SWE-bench, called SWE-bench-java. We have publicly
  released the dataset, along with the corresponding Docker-based evaluation
  environment and leaderboard, which will be continuously maintained and updated
  in the coming months. To verify the reliability of SWE-bench-java, we implement
  a classic method SWE-agent and test several powerful LLMs on it. As is well
  known, developing a high-quality multi-lingual benchmark is time-consuming and
  labor-intensive, so we welcome contributions through pull requests or
  collaboration to accelerate its iteration and refinement, paving the way for
  fully automated programming.
paper_authors._id: 66cd3ae3a7af45d8de3a6343
paper_authors.name: Daoguang Zan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-27T07:38:29.425Z
paper_authors._id: 61527edf8b55dbdae72874fa
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61527edf8b55dbdae72874fa/ZGWSBf_KSrDof6WyMoDMU.jpeg
paper_authors.fullname: Daoguang Zan
paper_authors.type: user
paper_authors.user: Daoguang
paper_authors._id: 66cd3ae3a7af45d8de3a6344
paper_authors.name: Zhirong Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-27T10:21:37.128Z
paper_authors._id: 658d32dfbe1495eb89635bfc
paper_authors.avatarUrl: /avatars/9493cfade840616dad627140dcb9e383.svg
paper_authors.fullname: huang zhirong
paper_authors.type: user
paper_authors.user: tonymgg
paper_authors._id: 66cd3ae3a7af45d8de3a6345
paper_authors.name: Ailun Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-27T10:21:43.217Z
paper_authors._id: 61b997ae22e5b0fdd501a0ec
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1639552935869-noauth.png
paper_authors.fullname: Ailun Yu
paper_authors.type: user
paper_authors.user: Tannpopo
paper_authors._id: 66cd3ae3a7af45d8de3a6346
paper_authors.name: Shaoxin Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-27T10:21:54.369Z
paper_authors._id: 64c8e32cbaa60107a6a543be
paper_authors.avatarUrl: /avatars/638dc910e4a4b525bfdadc601087a9c6.svg
paper_authors.fullname: linshaoxin
paper_authors.type: user
paper_authors.user: linshaoxin
paper_authors._id: 66cd3ae3a7af45d8de3a6347
paper_authors.name: Yifan Shi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-27T10:21:59.830Z
paper_authors._id: 66abada7c82d0b52b2a6b655
paper_authors.avatarUrl: /avatars/095f0e5e78bfb8c7fe0911b13b81ecce.svg
paper_authors.fullname: Yifan Shi
paper_authors.type: user
paper_authors.user: Yifannn
paper_authors._id: 66cd3ae3a7af45d8de3a6348
paper_authors.name: Wei Liu
paper_authors._id: 66cd3ae3a7af45d8de3a6349
paper_authors.name: Dong Chen
paper_authors._id: 66cd3ae3a7af45d8de3a634a
paper_authors.name: Zongshuai Qi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-27T10:22:30.764Z
paper_authors._id: 66cd4056332ce5c7468455da
paper_authors.avatarUrl: /avatars/bec2a903fcdf9299261cb50f5c978eb7.svg
paper_authors.fullname: Zongshuai Qi
paper_authors.type: user
paper_authors.user: kylin818
paper_authors._id: 66cd3ae3a7af45d8de3a634b
paper_authors.name: Hao Yu
paper_authors._id: 66cd3ae3a7af45d8de3a634c
paper_authors.name: Lei Yu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-29T07:23:37.502Z
paper_authors._id: 634786226f8773f2a28dfa5e
paper_authors.avatarUrl: /avatars/cdf5352e1346b1b61f2de9ec98abc96e.svg
paper_authors.fullname: yulei
paper_authors.type: user
paper_authors.user: yulei123aaa
paper_authors._id: 66cd3ae3a7af45d8de3a634d
paper_authors.name: Dezhi Ran
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-27T10:23:08.635Z
paper_authors._id: 63ba68ec0a9866b28cb2d53e
paper_authors.avatarUrl: /avatars/11cdb3f2cac4752cf430a5c98b0ae3a4.svg
paper_authors.fullname: Dezhi Ran
paper_authors.type: user
paper_authors.user: DezhiRan
paper_authors._id: 66cd3ae3a7af45d8de3a634e
paper_authors.name: Muhan Zeng
paper_authors._id: 66cd3ae3a7af45d8de3a634f
paper_authors.name: Bo Shen
paper_authors._id: 66cd3ae3a7af45d8de3a6350
paper_authors.name: Pan Bian
paper_authors._id: 66cd3ae3a7af45d8de3a6351
paper_authors.name: Guangtai Liang
paper_authors._id: 66cd3ae3a7af45d8de3a6352
paper_authors.name: Bei Guan
paper_authors._id: 66cd3ae3a7af45d8de3a6353
paper_authors.name: Pengjie Huang
paper_authors._id: 66cd3ae3a7af45d8de3a6354
paper_authors.name: Tao Xie
paper_authors._id: 66cd3ae3a7af45d8de3a6355
paper_authors.name: Yongji Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-27T10:42:51.683Z
paper_authors._id: 645c3d8a1d2ba8a1cf48683a
paper_authors.avatarUrl: /avatars/152eb17d5918edc051804b4cbc508c38.svg
paper_authors.fullname: Wang
paper_authors.type: user
paper_authors.user: Yongji
paper_authors._id: 66cd3ae3a7af45d8de3a6356
paper_authors.name: Qianxiang Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-27T10:42:33.977Z
paper_authors._id: 632db5eba0fd0d71dcc17290
paper_authors.avatarUrl: /avatars/6c7b096d495d1128e19d4fb18789e129.svg
paper_authors.fullname: Qianxiang Wang
paper_authors.type: user
paper_authors.user: WangQ
paper_title: SWE-bench-java: A GitHub Issue Resolving Benchmark for Java
paper_summary: GitHub issue resolving is a critical task in software engineering, recently
  gaining significant attention in both industry and academia. Within this task,
  SWE-bench has been released to evaluate issue resolving capabilities of large
  language models (LLMs), but has so far only focused on Python version. However,
  supporting more programming languages is also important, as there is a strong
  demand in industry. As a first step toward multilingual support, we have
  developed a Java version of SWE-bench, called SWE-bench-java. We have publicly
  released the dataset, along with the corresponding Docker-based evaluation
  environment and leaderboard, which will be continuously maintained and updated
  in the coming months. To verify the reliability of SWE-bench-java, we implement
  a classic method SWE-agent and test several powerful LLMs on it. As is well
  known, developing a high-quality multi-lingual benchmark is time-consuming and
  labor-intensive, so we welcome contributions through pull requests or
  collaboration to accelerate its iteration and refinement, paving the way for
  fully automated programming.