WebGen-Bench: Evaluating LLMs on Generating Interactive and Functional
  Websites from Scratch
LLM-based agents have demonstrated great potential in generating and managing
code within complex codebases. In this paper, we introduce WebGen-Bench, a
novel benchmark designed to measure an LLM-based agent's ability to create
multi-file website codebases from scratch. It contains diverse instructions for
website generation, created through the combined efforts of human annotators
and GPT-4o. These instructions span three major categories and thirteen minor
categories, encompassing nearly all important types of web applications. To
assess the quality of the generated websites, we use GPT-4o to generate test
cases targeting each functionality described in the instructions, and then
manually filter, adjust, and organize them to ensure accuracy, resulting in 647
test cases. Each test case specifies an operation to be performed on the
website and the expected result after the operation. To automate testing and
improve reproducibility, we employ a powerful web-navigation agent to execute
tests on the generated websites and determine whether the observed responses
align with the expected results. We evaluate three high-performance code-agent
frameworks, Bolt.diy, OpenHands, and Aider, using multiple proprietary and
open-source LLMs as engines. The best-performing combination, Bolt.diy powered
by DeepSeek-R1, achieves only 27.8\% accuracy on the test cases, highlighting
the challenging nature of our benchmark. Additionally, we construct
WebGen-Instruct, a training set consisting of 6,667 website-generation
instructions. Training Qwen2.5-Coder-32B-Instruct on Bolt.diy trajectories
generated from a subset of this training set achieves an accuracy of 38.2\%,
surpassing the performance of the best proprietary model.
681cb3dc815cbdd72910e310
Zimu Lu
claimed_verified
2025-05-13T08:13:03.384Z
64b0bfef2f2f9c345b87e673
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/7O3zCr_IyGvtz0BUdKNKi.jpeg
Zimu Lu
user
luzimu
681cb3dc815cbdd72910e311
Yunqiao Yang
admin_assigned
2025-05-13T08:36:05.709Z
66965c927e75dfb108c269b4
/avatars/3263fb0742a3d35fa19d6856d59701ee.svg
Yunqiao Yang
user
Yqy6
681cb3dc815cbdd72910e312
Houxing Ren
admin_assigned
2025-05-13T08:36:11.710Z
63e4b63dd6278c161be47f4b
https://cdn-avatars.huggingface.co/v1/production/uploads/63e4b63dd6278c161be47f4b/u0y1_o9WBqPJh8C8dimJ8.jpeg
Houxing Ren
user
Houxing
681cb3dc815cbdd72910e313
Haotian Hou
admin_assigned
2025-05-13T08:36:22.510Z
66cd8abad5026ff32afa0b95
/avatars/f0d0ad73d0f64a27f2aefe0de2608c44.svg
Hou Haotian
user
hht1113
681cb3dc815cbdd72910e314
Han Xiao
claimed_verified
2025-05-28T09:01:41.351Z
666aa99cd1652853e4f9a8b9
/avatars/7cd5a0c34b5ccb8eff5a353d88d15a93.svg
HanXiao
user
HanXiao1999
681cb3dc815cbdd72910e315
Ke Wang
681cb3dc815cbdd72910e316
Weikang Shi
admin_assigned
2025-05-13T08:36:43.939Z
6337e1d1107c4835a04c8607
/avatars/e85353b34c3da6efef84951934811fee.svg
Weikang Shi
user
shiwk20
681cb3dc815cbdd72910e317
Aojun Zhou
claimed_verified
2025-05-28T09:01:39.401Z
637de1520d5bb06fbe5207a9
/avatars/1090851217270c5a858b13e013356d4f.svg
AJ.Zhou
user
AJZhou
681cb3dc815cbdd72910e318
Mingjie Zhan
admin_assigned
2025-05-13T08:37:12.420Z
6301e46858f2b86bdfd49933
/avatars/d1c3fd94581f22a41506b24375a22bd3.svg
ZhanMingjie
user
Mingjie
681cb3dc815cbdd72910e319
Hongsheng Li
admin_assigned
2025-05-13T08:37:20.031Z
65c04e9c27a5fdca81abcbd9
/avatars/12a155683c824fa23da4a9e2bed4f64e.svg
Hongsheng LI
user
hsli-cuhk
WebGen-Bench: Evaluating LLMs on Generating Interactive and Functional
  Websites from Scratch
LLM-based agents have demonstrated great potential in generating and managing
code within complex codebases. In this paper, we introduce WebGen-Bench, a
novel benchmark designed to measure an LLM-based agent's ability to create
multi-file website codebases from scratch. It contains diverse instructions for
website generation, created through the combined efforts of human annotators
and GPT-4o. These instructions span three major categories and thirteen minor
categories, encompassing nearly all important types of web applications. To
assess the quality of the generated websites, we use GPT-4o to generate test
cases targeting each functionality described in the instructions, and then
manually filter, adjust, and organize them to ensure accuracy, resulting in 647
test cases. Each test case specifies an operation to be performed on the
website and the expected result after the operation. To automate testing and
improve reproducibility, we employ a powerful web-navigation agent to execute
tests on the generated websites and determine whether the observed responses
align with the expected results. We evaluate three high-performance code-agent
frameworks, Bolt.diy, OpenHands, and Aider, using multiple proprietary and
open-source LLMs as engines. The best-performing combination, Bolt.diy powered
by DeepSeek-R1, achieves only 27.8\% accuracy on the test cases, highlighting
the challenging nature of our benchmark. Additionally, we construct
WebGen-Instruct, a training set consisting of 6,667 website-generation
instructions. Training Qwen2.5-Coder-32B-Instruct on Bolt.diy trajectories
generated from a subset of this training set achieves an accuracy of 38.2\%,
surpassing the performance of the best proprietary model.
A benchmark named WebGen-Bench measures LLM-based agents' ability to generate multi-file website codebases, achieving higher accuracy with open-source models than proprietary ones.
LLM-based agents WebGen-Bench website codebases GPT-4 Bolt.diy OpenHands Aider DeepSeek-R1 Qwen2.5-Coder-32B-Instruct web-navigation agent website-generation instructions