title: WebGen-Bench: Evaluating LLMs on Generating Interactive and Functional
    Websites from Scratch
summary: LLM-based agents have demonstrated great potential in generating and managing
  code within complex codebases. In this paper, we introduce WebGen-Bench, a
  novel benchmark designed to measure an LLM-based agent's ability to create
  multi-file website codebases from scratch. It contains diverse instructions for
  website generation, created through the combined efforts of human annotators
  and GPT-4o. These instructions span three major categories and thirteen minor
  categories, encompassing nearly all important types of web applications. To
  assess the quality of the generated websites, we use GPT-4o to generate test
  cases targeting each functionality described in the instructions, and then
  manually filter, adjust, and organize them to ensure accuracy, resulting in 647
  test cases. Each test case specifies an operation to be performed on the
  website and the expected result after the operation. To automate testing and
  improve reproducibility, we employ a powerful web-navigation agent to execute
  tests on the generated websites and determine whether the observed responses
  align with the expected results. We evaluate three high-performance code-agent
  frameworks, Bolt.diy, OpenHands, and Aider, using multiple proprietary and
  open-source LLMs as engines. The best-performing combination, Bolt.diy powered
  by DeepSeek-R1, achieves only 27.8\% accuracy on the test cases, highlighting
  the challenging nature of our benchmark. Additionally, we construct
  WebGen-Instruct, a training set consisting of 6,667 website-generation
  instructions. Training Qwen2.5-Coder-32B-Instruct on Bolt.diy trajectories
  generated from a subset of this training set achieves an accuracy of 38.2\%,
  surpassing the performance of the best proprietary model.
paper_authors._id: 681cb3dc815cbdd72910e310
paper_authors.name: Zimu Lu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-13T08:13:03.384Z
paper_authors._id: 64b0bfef2f2f9c345b87e673
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/7O3zCr_IyGvtz0BUdKNKi.jpeg
paper_authors.fullname: Zimu Lu
paper_authors.type: user
paper_authors.user: luzimu
paper_authors._id: 681cb3dc815cbdd72910e311
paper_authors.name: Yunqiao Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-13T08:36:05.709Z
paper_authors._id: 66965c927e75dfb108c269b4
paper_authors.avatarUrl: /avatars/3263fb0742a3d35fa19d6856d59701ee.svg
paper_authors.fullname: Yunqiao Yang
paper_authors.type: user
paper_authors.user: Yqy6
paper_authors._id: 681cb3dc815cbdd72910e312
paper_authors.name: Houxing Ren
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-13T08:36:11.710Z
paper_authors._id: 63e4b63dd6278c161be47f4b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63e4b63dd6278c161be47f4b/u0y1_o9WBqPJh8C8dimJ8.jpeg
paper_authors.fullname: Houxing Ren
paper_authors.type: user
paper_authors.user: Houxing
paper_authors._id: 681cb3dc815cbdd72910e313
paper_authors.name: Haotian Hou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-13T08:36:22.510Z
paper_authors._id: 66cd8abad5026ff32afa0b95
paper_authors.avatarUrl: /avatars/f0d0ad73d0f64a27f2aefe0de2608c44.svg
paper_authors.fullname: Hou Haotian
paper_authors.type: user
paper_authors.user: hht1113
paper_authors._id: 681cb3dc815cbdd72910e314
paper_authors.name: Han Xiao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-28T09:01:41.351Z
paper_authors._id: 666aa99cd1652853e4f9a8b9
paper_authors.avatarUrl: /avatars/7cd5a0c34b5ccb8eff5a353d88d15a93.svg
paper_authors.fullname: HanXiao
paper_authors.type: user
paper_authors.user: HanXiao1999
paper_authors._id: 681cb3dc815cbdd72910e315
paper_authors.name: Ke Wang
paper_authors._id: 681cb3dc815cbdd72910e316
paper_authors.name: Weikang Shi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-13T08:36:43.939Z
paper_authors._id: 6337e1d1107c4835a04c8607
paper_authors.avatarUrl: /avatars/e85353b34c3da6efef84951934811fee.svg
paper_authors.fullname: Weikang Shi
paper_authors.type: user
paper_authors.user: shiwk20
paper_authors._id: 681cb3dc815cbdd72910e317
paper_authors.name: Aojun Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-28T09:01:39.401Z
paper_authors._id: 637de1520d5bb06fbe5207a9
paper_authors.avatarUrl: /avatars/1090851217270c5a858b13e013356d4f.svg
paper_authors.fullname: AJ.Zhou
paper_authors.type: user
paper_authors.user: AJZhou
paper_authors._id: 681cb3dc815cbdd72910e318
paper_authors.name: Mingjie Zhan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-13T08:37:12.420Z
paper_authors._id: 6301e46858f2b86bdfd49933
paper_authors.avatarUrl: /avatars/d1c3fd94581f22a41506b24375a22bd3.svg
paper_authors.fullname: ZhanMingjie
paper_authors.type: user
paper_authors.user: Mingjie
paper_authors._id: 681cb3dc815cbdd72910e319
paper_authors.name: Hongsheng Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-13T08:37:20.031Z
paper_authors._id: 65c04e9c27a5fdca81abcbd9
paper_authors.avatarUrl: /avatars/12a155683c824fa23da4a9e2bed4f64e.svg
paper_authors.fullname: Hongsheng LI
paper_authors.type: user
paper_authors.user: hsli-cuhk
paper_title: WebGen-Bench: Evaluating LLMs on Generating Interactive and Functional
    Websites from Scratch
paper_summary: LLM-based agents have demonstrated great potential in generating and managing
  code within complex codebases. In this paper, we introduce WebGen-Bench, a
  novel benchmark designed to measure an LLM-based agent's ability to create
  multi-file website codebases from scratch. It contains diverse instructions for
  website generation, created through the combined efforts of human annotators
  and GPT-4o. These instructions span three major categories and thirteen minor
  categories, encompassing nearly all important types of web applications. To
  assess the quality of the generated websites, we use GPT-4o to generate test
  cases targeting each functionality described in the instructions, and then
  manually filter, adjust, and organize them to ensure accuracy, resulting in 647
  test cases. Each test case specifies an operation to be performed on the
  website and the expected result after the operation. To automate testing and
  improve reproducibility, we employ a powerful web-navigation agent to execute
  tests on the generated websites and determine whether the observed responses
  align with the expected results. We evaluate three high-performance code-agent
  frameworks, Bolt.diy, OpenHands, and Aider, using multiple proprietary and
  open-source LLMs as engines. The best-performing combination, Bolt.diy powered
  by DeepSeek-R1, achieves only 27.8\% accuracy on the test cases, highlighting
  the challenging nature of our benchmark. Additionally, we construct
  WebGen-Instruct, a training set consisting of 6,667 website-generation
  instructions. Training Qwen2.5-Coder-32B-Instruct on Bolt.diy trajectories
  generated from a subset of this training set achieves an accuracy of 38.2\%,
  surpassing the performance of the best proprietary model.
paper_ai_summary: A benchmark named WebGen-Bench measures LLM-based agents' ability to generate multi-file website codebases, achieving higher accuracy with open-source models than proprietary ones.
paper_ai_keywords: LLM-based agents WebGen-Bench website codebases GPT-4 Bolt.diy OpenHands Aider DeepSeek-R1 Qwen2.5-Coder-32B-Instruct web-navigation agent website-generation instructions