Can AI Freelancers Compete? Benchmarking Earnings, Reliability, and Task
  Success at Scale
This study explores Large Language Models (LLMs) as autonomous agents for
real-world tasks, including freelance software development. This work presents
a new benchmark that evaluates LLMs on freelance programming and data analysis
tasks derived from economic data. We construct the benchmark using synthetic
tasks created from a Kaggle Freelancer dataset of job postings, with all job
prices standardized to USD (median fixed-project price around 250, and an
average of 306). Each task is accompanied by structured input-output test
cases and an estimated price tag, enabling automated correctness checking and a
monetary performance valuation. This approach is inspired by OpenAI's recent
SWE-Lancer benchmark (1,400 real Upwork tasks worth 1M total). Still, our
framework simplifies evaluation using programmatically testable tasks and
predicted price values, making it highly scalable and repeatable. On this
benchmark, we evaluate four modern LLMs - Claude 3.5 Haiku, GPT-4o-mini, Qwen
2.5, and Mistral. We report each model's accuracy (task success rate and
test-case pass rate) and the total "freelance earnings" it achieves (sum of
prices of solved tasks). Our results show that Claude 3.5 Haiku performs best,
earning approximately 1.52 million USD, followed closely by GPT-4o-mini at
1.49 million, then Qwen 2.5 (1.33M) and Mistral ($0.70M). We analyze the
distribution of errors per task and observe that the strongest models solve the
most tasks and rarely fail completely on any project. We discuss the
implications of these results for the feasibility of AI as a freelance
developer, the advantages and limitations of our automated benchmark approach,
and the gap between performance on structured tasks versus the true complexity
of real-world freelance jobs.
682d28e9ebd378a24654075d
David Noever
682d28e9ebd378a24654075e
Forrest McKee
Can AI Freelancers Compete? Benchmarking Earnings, Reliability, and Task
  Success at Scale
This study explores Large Language Models (LLMs) as autonomous agents for
real-world tasks, including freelance software development. This work presents
a new benchmark that evaluates LLMs on freelance programming and data analysis
tasks derived from economic data. We construct the benchmark using synthetic
tasks created from a Kaggle Freelancer dataset of job postings, with all job
prices standardized to USD (median fixed-project price around 250, and an
average of 306). Each task is accompanied by structured input-output test
cases and an estimated price tag, enabling automated correctness checking and a
monetary performance valuation. This approach is inspired by OpenAI's recent
SWE-Lancer benchmark (1,400 real Upwork tasks worth 1M total). Still, our
framework simplifies evaluation using programmatically testable tasks and
predicted price values, making it highly scalable and repeatable. On this
benchmark, we evaluate four modern LLMs - Claude 3.5 Haiku, GPT-4o-mini, Qwen
2.5, and Mistral. We report each model's accuracy (task success rate and
test-case pass rate) and the total "freelance earnings" it achieves (sum of
prices of solved tasks). Our results show that Claude 3.5 Haiku performs best,
earning approximately 1.52 million USD, followed closely by GPT-4o-mini at
1.49 million, then Qwen 2.5 (1.33M) and Mistral ($0.70M). We analyze the
distribution of errors per task and observe that the strongest models solve the
most tasks and rarely fail completely on any project. We discuss the
implications of these results for the feasibility of AI as a freelance
developer, the advantages and limitations of our automated benchmark approach,
and the gap between performance on structured tasks versus the true complexity
of real-world freelance jobs.
A new benchmark evaluates large language models on freelance programming and data analysis tasks, providing insights into their performance and feasibility as autonomous agents.
Large Language Models LLMs autonomous agents freelance programming data analysis benchmark synthetic tasks Kaggle Freelancer dataset structured input-output test cases OpenAI's SWE-Lancer benchmark programmatically testable tasks predicted price values task success rate test-case pass rate freelance earnings Claude 3.5 Haiku GPT-4o-mini Qwen 2.5 Mistral error distribution real-world freelance jobs