ArtifactsBench: Bridging the Visual-Interactive Gap in LLM Code
  Generation Evaluation
The generative capabilities of Large Language Models (LLMs) are rapidly
expanding from static code to dynamic, interactive visual artifacts. This
progress is bottlenecked by a critical evaluation gap: established benchmarks
focus on algorithmic correctness and are blind to the visual fidelity and
interactive integrity that define modern user experiences. To bridge this gap,
we introduce ArtifactsBench, a new benchmark and paradigm for the automated,
multimodal evaluation of visual code generation. Our framework programmatically
renders each generated artifact and captures its dynamic behavior through
temporal screenshots. This visual evidence, alongside the source code, is then
assessed by a Multimodal LLM (MLLM)-as-Judge, which is rigorously guided by a
fine-grained, per-task checklist to ensure holistic and reproducible scoring.
We construct a new benchmark of 1,825 diverse tasks and evaluate over 30
leading LLMs. Our automated evaluation achieves a striking 94.4% ranking
consistency with WebDev Arena, the gold-standard for human preference in web
development, and over 90% pairwise agreement with human experts. This
establishes ArtifactsBench as the first framework to reliably automate the
assessment of human-perceived quality at scale. Our analysis provides a
high-resolution map of the current SOTA, revealing that generalist models often
outperform domain-specific ones. We open-source ArtifactsBench, including the
benchmark, evaluation harness, and baseline results at
https://artifactsbenchmark.github.io/, to provide the community with a scalable
and accurate tool to accelerate the development of user-centric generative
models.
686c8487364e2ad167eb5386
Chenchen Zhang
686c8487364e2ad167eb5387
Yuhang Li
686c8487364e2ad167eb5388
Can Xu
686c8487364e2ad167eb5389
Jiaheng Liu
686c8487364e2ad167eb538a
Ao Liu
686c8487364e2ad167eb538b
Shihui Hu
686c8487364e2ad167eb538c
Dengpeng Wu
686c8487364e2ad167eb538d
Guanhua Huang
686c8487364e2ad167eb538e
Kejiao Li
686c8487364e2ad167eb538f
Qi Yi
686c8487364e2ad167eb5390
Ruibin Xiong
686c8487364e2ad167eb5391
Haotian Zhu
686c8487364e2ad167eb5392
Yuanxing Zhang
686c8487364e2ad167eb5393
Yuhao Jiang
686c8487364e2ad167eb5394
Yue Zhang
686c8487364e2ad167eb5395
Zenan Xu
686c8487364e2ad167eb5396
Bohui Zhai
686c8487364e2ad167eb5397
Guoxiang He
686c8487364e2ad167eb5398
Hebin Li
686c8487364e2ad167eb5399
Jie Zhao
686c8487364e2ad167eb539a
Le Zhang
686c8487364e2ad167eb539b
Lingyun Tan
686c8487364e2ad167eb539c
Pengyu Guo
686c8487364e2ad167eb539d
Xianshu Pang
686c8487364e2ad167eb539e
Yang Ruan
686c8487364e2ad167eb539f
Zhifeng Zhang
686c8487364e2ad167eb53a0
Zhonghu Wang
686c8487364e2ad167eb53a1
Ziyan Xu
686c8487364e2ad167eb53a2
Zuopu Yin
686c8487364e2ad167eb53a3
Wiggin Zhou
686c8487364e2ad167eb53a4
Chayse Zhou
686c8487364e2ad167eb53a5
Fengzong Lian
ArtifactsBench: Bridging the Visual-Interactive Gap in LLM Code
  Generation Evaluation
The generative capabilities of Large Language Models (LLMs) are rapidly
expanding from static code to dynamic, interactive visual artifacts. This
progress is bottlenecked by a critical evaluation gap: established benchmarks
focus on algorithmic correctness and are blind to the visual fidelity and
interactive integrity that define modern user experiences. To bridge this gap,
we introduce ArtifactsBench, a new benchmark and paradigm for the automated,
multimodal evaluation of visual code generation. Our framework programmatically
renders each generated artifact and captures its dynamic behavior through
temporal screenshots. This visual evidence, alongside the source code, is then
assessed by a Multimodal LLM (MLLM)-as-Judge, which is rigorously guided by a
fine-grained, per-task checklist to ensure holistic and reproducible scoring.
We construct a new benchmark of 1,825 diverse tasks and evaluate over 30
leading LLMs. Our automated evaluation achieves a striking 94.4% ranking
consistency with WebDev Arena, the gold-standard for human preference in web
development, and over 90% pairwise agreement with human experts. This
establishes ArtifactsBench as the first framework to reliably automate the
assessment of human-perceived quality at scale. Our analysis provides a
high-resolution map of the current SOTA, revealing that generalist models often
outperform domain-specific ones. We open-source ArtifactsBench, including the
benchmark, evaluation harness, and baseline results at
https://artifactsbenchmark.github.io/, to provide the community with a scalable
and accurate tool to accelerate the development of user-centric generative
models.
ArtifactsBench is a new benchmark for evaluating the visual and interactive quality of code-generated artifacts using a Multimodal LLM-as-Judge, achieving high consistency with human preferences.
Large Language Models LLMs ArtifactsBench Multimodal LLM MLLM visual code generation temporal screenshots WebDev Arena human-perceived quality generalist models domain-specific models