title: GSO: Challenging Software Optimization Tasks for Evaluating SWE-Agents
summary: Developing high-performance software is a complex task that requires
  specialized expertise. We introduce GSO, a benchmark for evaluating language
  models' capabilities in developing high-performance software. We develop an
  automated pipeline that generates and executes performance tests to analyze
  repository commit histories to identify 102 challenging optimization tasks
  across 10 codebases, spanning diverse domains and programming languages. An
  agent is provided with a codebase and performance test as a precise
  specification, and tasked to improve the runtime efficiency, which is measured
  against the expert developer optimization. Our quantitative evaluation reveals
  that leading SWE-Agents struggle significantly, achieving less than 5% success
  rate, with limited improvements even with inference-time scaling. Our
  qualitative analysis identifies key failure modes, including difficulties with
  low-level languages, practicing lazy optimization strategies, and challenges in
  accurately localizing bottlenecks. We release the code and artifacts of our
  benchmark along with agent trajectories to enable future research.
paper_authors._id: 6839ee8e011f06eb0e6ec2c8
paper_authors.name: Manish Shetty
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-02T07:43:47.830Z
paper_authors._id: 637bdc36e073a16fb7886923
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/637bdc36e073a16fb7886923/jaeT_CK3TdidkrZYado9N.jpeg
paper_authors.fullname: Manish Shetty
paper_authors.type: user
paper_authors.user: manishs
paper_authors._id: 6839ee8e011f06eb0e6ec2c9
paper_authors.name: Naman Jain
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-02T07:43:50.728Z
paper_authors._id: 6411ed451e42164b9f0f8b24
paper_authors.avatarUrl: /avatars/a5c023f407894d3d1dba7b343b847380.svg
paper_authors.fullname: Naman Jain
paper_authors.type: user
paper_authors.user: StringChaos
paper_authors._id: 6839ee8e011f06eb0e6ec2ca
paper_authors.name: Jinjian Liu
paper_authors._id: 6839ee8e011f06eb0e6ec2cb
paper_authors.name: Vijay Kethanaboyina
paper_authors._id: 6839ee8e011f06eb0e6ec2cc
paper_authors.name: Koushik Sen
paper_authors._id: 6839ee8e011f06eb0e6ec2cd
paper_authors.name: Ion Stoica
paper_title: GSO: Challenging Software Optimization Tasks for Evaluating SWE-Agents
paper_summary: Developing high-performance software is a complex task that requires
  specialized expertise. We introduce GSO, a benchmark for evaluating language
  models' capabilities in developing high-performance software. We develop an
  automated pipeline that generates and executes performance tests to analyze
  repository commit histories to identify 102 challenging optimization tasks
  across 10 codebases, spanning diverse domains and programming languages. An
  agent is provided with a codebase and performance test as a precise
  specification, and tasked to improve the runtime efficiency, which is measured
  against the expert developer optimization. Our quantitative evaluation reveals
  that leading SWE-Agents struggle significantly, achieving less than 5% success
  rate, with limited improvements even with inference-time scaling. Our
  qualitative analysis identifies key failure modes, including difficulties with
  low-level languages, practicing lazy optimization strategies, and challenges in
  accurately localizing bottlenecks. We release the code and artifacts of our
  benchmark along with agent trajectories to enable future research.
paper_ai_summary: A benchmark evaluates high-performance software development capabilities of language models, identifying significant challenges and failure modes.
paper_ai_keywords: GSO SWE-Agents performance tests repository commit histories optimization tasks runtime efficiency expert developer optimization low-level languages lazy optimization strategies bottleneck localization