SWE-Perf: Can Language Models Optimize Code Performance on Real-World
  Repositories?
Code performance optimization is paramount in real-world software engineering
and critical for production-level systems. While Large Language Models (LLMs)
have demonstrated impressive capabilities in code generation and bug fixing,
their proficiency in enhancing code performance at the repository level remains
largely unexplored. To address this gap, we introduce SWE-Perf, the first
benchmark specifically designed to systematically evaluate LLMs on code
performance optimization tasks within authentic repository contexts. SWE-Perf
comprises 140 carefully curated instances, each derived from
performance-improving pull requests from popular GitHub repositories. Each
benchmark instance includes the relevant codebase, target functions,
performance-related tests, expert-authored patches, and executable
environments. Through a comprehensive evaluation of representative methods that
span file-level and repo-level approaches (e.g., Agentless and OpenHands), we
reveal a substantial capability gap between existing LLMs and expert-level
optimization performance, highlighting critical research opportunities in this
emerging field.
68788b9b001546c83aa4f9ed
Xinyi He
claimed_verified
2025-07-17T18:43:15.979Z
65008c1b7aa4d88f207bedc5
https://cdn-avatars.huggingface.co/v1/production/uploads/65008c1b7aa4d88f207bedc5/e4YN_z1UyV0M47nezcFOx.jpeg
Xinyi He
user
betty12
68788b9b001546c83aa4f9ee
Qian Liu
claimed_verified
2025-07-17T08:17:46.375Z
612ee6a7b960e78c6d2319d4
https://cdn-avatars.huggingface.co/v1/production/uploads/612ee6a7b960e78c6d2319d4/2Hu9BaAyXbyh1vt0v1Qui.jpeg
Qian Liu
user
SivilTaram
68788b9b001546c83aa4f9ef
Mingzhe Du
claimed_verified
2025-07-17T09:08:30.259Z
61711f02e0b1ddb56eb9b526
/avatars/3e2fdf774f5bc1f73b450486d6da42d4.svg
Mingzhe Du
user
Elfsong
68788b9b001546c83aa4f9f0
Lin Yan
68788b9b001546c83aa4f9f1
Zhijie Fan
68788b9b001546c83aa4f9f2
Yiming Huang
68788b9b001546c83aa4f9f3
Zejian Yuan
68788b9b001546c83aa4f9f4
Zejun Ma
SWE-Perf: Can Language Models Optimize Code Performance on Real-World
  Repositories?
Code performance optimization is paramount in real-world software engineering
and critical for production-level systems. While Large Language Models (LLMs)
have demonstrated impressive capabilities in code generation and bug fixing,
their proficiency in enhancing code performance at the repository level remains
largely unexplored. To address this gap, we introduce SWE-Perf, the first
benchmark specifically designed to systematically evaluate LLMs on code
performance optimization tasks within authentic repository contexts. SWE-Perf
comprises 140 carefully curated instances, each derived from
performance-improving pull requests from popular GitHub repositories. Each
benchmark instance includes the relevant codebase, target functions,
performance-related tests, expert-authored patches, and executable
environments. Through a comprehensive evaluation of representative methods that
span file-level and repo-level approaches (e.g., Agentless and OpenHands), we
reveal a substantial capability gap between existing LLMs and expert-level
optimization performance, highlighting critical research opportunities in this
emerging field.
SWE-Perf is a benchmark for evaluating LLMs in code performance optimization using real-world repository data, revealing significant gaps compared to expert performance.
Large Language Models LLMs code performance optimization benchmark performance-improving pull requests codebase target functions performance-related tests expert-authored patches executable environments Agentless OpenHands