title: OmniGIRL: A Multilingual and Multimodal Benchmark for GitHub Issue
    Resolution
summary: The GitHub issue resolution task aims to resolve issues reported in
  repositories automatically. With advances in large language models (LLMs), this
  task has gained increasing attention, and several benchmarks are proposed to
  evaluate the issue resolution ability of LLMs. However, existing benchmarks
  have three main limitations. First, current benchmarks focus on a single
  programming language, limiting the evaluation of issues from repositories
  across different languages. Second, they usually cover a narrow range of
  domains, which may fail to represent the diversity of real-world issues. Third,
  existing benchmarks rely solely on textual information in issue descriptions,
  overlooking multimodal information such as images in issues. In this paper, we
  propose OmniGIRL, a GitHub Issue ResoLution benchmark that is multilingual,
  multimodal, and multi-domain. OmniGIRL includes 959 task instances, which are
  collected from repositories across four programming languages (i.e., Python,
  JavaScript, TypeScript, and Java) and eight different domains. Our evaluation
  shows that current LLMs show limited performances on OmniGIRL. Notably, the
  best-performing model, GPT-4o, resolves only 8.6% of the issues. Besides, we
  find that current LLMs struggle to resolve issues requiring understanding
  images. The best performance is achieved by Claude-3.5-Sonnet, which resolves
  only 10.5% of the issues with image information. Finally, we analyze the
  reasons behind current LLMs' failure on OmniGIRL, providing insights for future
  improvements.
paper_authors._id: 681c7a87215f1831099fdfd2
paper_authors.name: Lianghong Guo
paper_authors._id: 681c7a87215f1831099fdfd3
paper_authors.name: Wei Tao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-08T09:58:00.562Z
paper_authors._id: 6355473d525beaee688b7ba1
paper_authors.avatarUrl: /avatars/0a0f0acc65829c6d864440444c580698.svg
paper_authors.fullname: Wei Tao
paper_authors.type: user
paper_authors.user: itaowe
paper_authors._id: 681c7a87215f1831099fdfd4
paper_authors.name: Runhan Jiang
paper_authors._id: 681c7a87215f1831099fdfd5
paper_authors.name: Yanlin Wang
paper_authors._id: 681c7a87215f1831099fdfd6
paper_authors.name: Jiachi Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-14T13:00:45.102Z
paper_authors._id: 66ee1c302e68abec6890ac35
paper_authors.avatarUrl: /avatars/c1130a51d58591edec23f10bf0291187.svg
paper_authors.fullname: Chen
paper_authors.type: user
paper_authors.user: Jiachi317
paper_authors._id: 681c7a87215f1831099fdfd7
paper_authors.name: Xilin Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-14T13:00:55.524Z
paper_authors._id: 65efafff61b46846709541c0
paper_authors.avatarUrl: /avatars/a81e7c5d517cd089b9fbd7c10fc23250.svg
paper_authors.fullname: Liu Xilin
paper_authors.type: user
paper_authors.user: armstrong111
paper_authors._id: 681c7a87215f1831099fdfd8
paper_authors.name: Yuchi Ma
paper_authors._id: 681c7a87215f1831099fdfd9
paper_authors.name: Mingzhi Mao
paper_authors._id: 681c7a87215f1831099fdfda
paper_authors.name: Hongyu Zhang
paper_authors._id: 681c7a87215f1831099fdfdb
paper_authors.name: Zibin Zheng
paper_title: OmniGIRL: A Multilingual and Multimodal Benchmark for GitHub Issue
    Resolution
paper_summary: The GitHub issue resolution task aims to resolve issues reported in
  repositories automatically. With advances in large language models (LLMs), this
  task has gained increasing attention, and several benchmarks are proposed to
  evaluate the issue resolution ability of LLMs. However, existing benchmarks
  have three main limitations. First, current benchmarks focus on a single
  programming language, limiting the evaluation of issues from repositories
  across different languages. Second, they usually cover a narrow range of
  domains, which may fail to represent the diversity of real-world issues. Third,
  existing benchmarks rely solely on textual information in issue descriptions,
  overlooking multimodal information such as images in issues. In this paper, we
  propose OmniGIRL, a GitHub Issue ResoLution benchmark that is multilingual,
  multimodal, and multi-domain. OmniGIRL includes 959 task instances, which are
  collected from repositories across four programming languages (i.e., Python,
  JavaScript, TypeScript, and Java) and eight different domains. Our evaluation
  shows that current LLMs show limited performances on OmniGIRL. Notably, the
  best-performing model, GPT-4o, resolves only 8.6% of the issues. Besides, we
  find that current LLMs struggle to resolve issues requiring understanding
  images. The best performance is achieved by Claude-3.5-Sonnet, which resolves
  only 10.5% of the issues with image information. Finally, we analyze the
  reasons behind current LLMs' failure on OmniGIRL, providing insights for future
  improvements.
paper_ai_summary: OmniGIRL, a multilingual, multimodal, and multi-domain GitHub issue resolution benchmark, demonstrates that current LLMs perform poorly, particularly in issues involving images.
paper_ai_keywords: multilingual multimodal multi-domain GitHub issue resolution benchmark programming languages domains large language models