title: SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance
    Software Engineering?
summary: We introduce SWE-Lancer, a benchmark of over 1,400 freelance software
  engineering tasks from Upwork, valued at \1 million USD total in real-world
  payouts. SWE-Lancer encompasses both independent engineering tasks--ranging
  from 50 bug fixes to \$32,000 feature implementations--and managerial tasks,
  where models choose between technical implementation proposals. Independent
  tasks are graded with end-to-end tests triple-verified by experienced software
  engineers, while managerial decisions are assessed against the choices of the
  original hired engineering managers. We evaluate model performance and find
  that frontier models are still unable to solve the majority of tasks. To
  facilitate future research, we open-source a unified Docker image and a public
  evaluation split, SWE-Lancer Diamond
  (https://github.com/openai/SWELancer-Benchmark). By mapping model performance
  to monetary value, we hope SWE-Lancer enables greater research into the
  economic impact of AI model development.
paper_authors._id: 67b41a72a38d04cc6148d80e
paper_authors.name: Samuel Miserendino
paper_authors._id: 67b41a72a38d04cc6148d80f
paper_authors.name: Michele Wang
paper_authors._id: 67b41a72a38d04cc6148d810
paper_authors.name: Tejal Patwardhan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T15:15:20.891Z
paper_authors._id: 64d4f504887f55fb6eedec74
paper_authors.avatarUrl: /avatars/054fb826890adcb330f0e4cbca3ef7c4.svg
paper_authors.fullname: Tejal Patwardhan
paper_authors.type: user
paper_authors.user: tejalp
paper_authors._id: 67b41a72a38d04cc6148d811
paper_authors.name: Johannes Heidecke
paper_title: SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance
    Software Engineering?
paper_summary: We introduce SWE-Lancer, a benchmark of over 1,400 freelance software
  engineering tasks from Upwork, valued at \1 million USD total in real-world
  payouts. SWE-Lancer encompasses both independent engineering tasks--ranging
  from 50 bug fixes to \$32,000 feature implementations--and managerial tasks,
  where models choose between technical implementation proposals. Independent
  tasks are graded with end-to-end tests triple-verified by experienced software
  engineers, while managerial decisions are assessed against the choices of the
  original hired engineering managers. We evaluate model performance and find
  that frontier models are still unable to solve the majority of tasks. To
  facilitate future research, we open-source a unified Docker image and a public
  evaluation split, SWE-Lancer Diamond
  (https://github.com/openai/SWELancer-Benchmark). By mapping model performance
  to monetary value, we hope SWE-Lancer enables greater research into the
  economic impact of AI model development.