title: Can AI Freelancers Compete? Benchmarking Earnings, Reliability, and Task
    Success at Scale
summary: This study explores Large Language Models (LLMs) as autonomous agents for
  real-world tasks, including freelance software development. This work presents
  a new benchmark that evaluates LLMs on freelance programming and data analysis
  tasks derived from economic data. We construct the benchmark using synthetic
  tasks created from a Kaggle Freelancer dataset of job postings, with all job
  prices standardized to USD (median fixed-project price around 250, and an
  average of 306). Each task is accompanied by structured input-output test
  cases and an estimated price tag, enabling automated correctness checking and a
  monetary performance valuation. This approach is inspired by OpenAI's recent
  SWE-Lancer benchmark (1,400 real Upwork tasks worth 1M total). Still, our
  framework simplifies evaluation using programmatically testable tasks and
  predicted price values, making it highly scalable and repeatable. On this
  benchmark, we evaluate four modern LLMs - Claude 3.5 Haiku, GPT-4o-mini, Qwen
  2.5, and Mistral. We report each model's accuracy (task success rate and
  test-case pass rate) and the total "freelance earnings" it achieves (sum of
  prices of solved tasks). Our results show that Claude 3.5 Haiku performs best,
  earning approximately 1.52 million USD, followed closely by GPT-4o-mini at
  1.49 million, then Qwen 2.5 (1.33M) and Mistral ($0.70M). We analyze the
  distribution of errors per task and observe that the strongest models solve the
  most tasks and rarely fail completely on any project. We discuss the
  implications of these results for the feasibility of AI as a freelance
  developer, the advantages and limitations of our automated benchmark approach,
  and the gap between performance on structured tasks versus the true complexity
  of real-world freelance jobs.
paper_authors._id: 682d28e9ebd378a24654075d
paper_authors.name: David Noever
paper_authors._id: 682d28e9ebd378a24654075e
paper_authors.name: Forrest McKee
paper_title: Can AI Freelancers Compete? Benchmarking Earnings, Reliability, and Task
    Success at Scale
paper_summary: This study explores Large Language Models (LLMs) as autonomous agents for
  real-world tasks, including freelance software development. This work presents
  a new benchmark that evaluates LLMs on freelance programming and data analysis
  tasks derived from economic data. We construct the benchmark using synthetic
  tasks created from a Kaggle Freelancer dataset of job postings, with all job
  prices standardized to USD (median fixed-project price around 250, and an
  average of 306). Each task is accompanied by structured input-output test
  cases and an estimated price tag, enabling automated correctness checking and a
  monetary performance valuation. This approach is inspired by OpenAI's recent
  SWE-Lancer benchmark (1,400 real Upwork tasks worth 1M total). Still, our
  framework simplifies evaluation using programmatically testable tasks and
  predicted price values, making it highly scalable and repeatable. On this
  benchmark, we evaluate four modern LLMs - Claude 3.5 Haiku, GPT-4o-mini, Qwen
  2.5, and Mistral. We report each model's accuracy (task success rate and
  test-case pass rate) and the total "freelance earnings" it achieves (sum of
  prices of solved tasks). Our results show that Claude 3.5 Haiku performs best,
  earning approximately 1.52 million USD, followed closely by GPT-4o-mini at
  1.49 million, then Qwen 2.5 (1.33M) and Mistral ($0.70M). We analyze the
  distribution of errors per task and observe that the strongest models solve the
  most tasks and rarely fail completely on any project. We discuss the
  implications of these results for the feasibility of AI as a freelance
  developer, the advantages and limitations of our automated benchmark approach,
  and the gap between performance on structured tasks versus the true complexity
  of real-world freelance jobs.
paper_ai_summary: A new benchmark evaluates large language models on freelance programming and data analysis tasks, providing insights into their performance and feasibility as autonomous agents.
paper_ai_keywords: Large Language Models LLMs autonomous agents freelance programming data analysis benchmark synthetic tasks Kaggle Freelancer dataset structured input-output test cases OpenAI's SWE-Lancer benchmark programmatically testable tasks predicted price values task success rate test-case pass rate freelance earnings Claude 3.5 Haiku GPT-4o-mini Qwen 2.5 Mistral error distribution real-world freelance jobs