GSO: Challenging Software Optimization Tasks for Evaluating SWE-Agents
Developing high-performance software is a complex task that requires
specialized expertise. We introduce GSO, a benchmark for evaluating language
models' capabilities in developing high-performance software. We develop an
automated pipeline that generates and executes performance tests to analyze
repository commit histories to identify 102 challenging optimization tasks
across 10 codebases, spanning diverse domains and programming languages. An
agent is provided with a codebase and performance test as a precise
specification, and tasked to improve the runtime efficiency, which is measured
against the expert developer optimization. Our quantitative evaluation reveals
that leading SWE-Agents struggle significantly, achieving less than 5% success
rate, with limited improvements even with inference-time scaling. Our
qualitative analysis identifies key failure modes, including difficulties with
low-level languages, practicing lazy optimization strategies, and challenges in
accurately localizing bottlenecks. We release the code and artifacts of our
benchmark along with agent trajectories to enable future research.
6839ee8e011f06eb0e6ec2c8
Manish Shetty
claimed_verified
2025-06-02T07:43:47.830Z
637bdc36e073a16fb7886923
https://cdn-avatars.huggingface.co/v1/production/uploads/637bdc36e073a16fb7886923/jaeT_CK3TdidkrZYado9N.jpeg
Manish Shetty
user
manishs
6839ee8e011f06eb0e6ec2c9
Naman Jain
claimed_verified
2025-06-02T07:43:50.728Z
6411ed451e42164b9f0f8b24
/avatars/a5c023f407894d3d1dba7b343b847380.svg
Naman Jain
user
StringChaos
6839ee8e011f06eb0e6ec2ca
Jinjian Liu
6839ee8e011f06eb0e6ec2cb
Vijay Kethanaboyina
6839ee8e011f06eb0e6ec2cc
Koushik Sen
6839ee8e011f06eb0e6ec2cd
Ion Stoica
GSO: Challenging Software Optimization Tasks for Evaluating SWE-Agents
Developing high-performance software is a complex task that requires
specialized expertise. We introduce GSO, a benchmark for evaluating language
models' capabilities in developing high-performance software. We develop an
automated pipeline that generates and executes performance tests to analyze
repository commit histories to identify 102 challenging optimization tasks
across 10 codebases, spanning diverse domains and programming languages. An
agent is provided with a codebase and performance test as a precise
specification, and tasked to improve the runtime efficiency, which is measured
against the expert developer optimization. Our quantitative evaluation reveals
that leading SWE-Agents struggle significantly, achieving less than 5% success
rate, with limited improvements even with inference-time scaling. Our
qualitative analysis identifies key failure modes, including difficulties with
low-level languages, practicing lazy optimization strategies, and challenges in
accurately localizing bottlenecks. We release the code and artifacts of our
benchmark along with agent trajectories to enable future research.
A benchmark evaluates high-performance software development capabilities of language models, identifying significant challenges and failure modes.
GSO SWE-Agents performance tests repository commit histories optimization tasks runtime efficiency expert developer optimization low-level languages lazy optimization strategies bottleneck localization