WebNovelBench: Placing LLM Novelists on the Web Novel Distribution
Robustly evaluating the long-form storytelling capabilities of Large Language
Models (LLMs) remains a significant challenge, as existing benchmarks often
lack the necessary scale, diversity, or objective measures. To address this, we
introduce WebNovelBench, a novel benchmark specifically designed for evaluating
long-form novel generation. WebNovelBench leverages a large-scale dataset of
over 4,000 Chinese web novels, framing evaluation as a synopsis-to-story
generation task. We propose a multi-faceted framework encompassing eight
narrative quality dimensions, assessed automatically via an LLM-as-Judge
approach. Scores are aggregated using Principal Component Analysis and mapped
to a percentile rank against human-authored works. Our experiments demonstrate
that WebNovelBench effectively differentiates between human-written
masterpieces, popular web novels, and LLM-generated content. We provide a
comprehensive analysis of 24 state-of-the-art LLMs, ranking their storytelling
abilities and offering insights for future development. This benchmark provides
a scalable, replicable, and data-driven methodology for assessing and advancing
LLM-driven narrative generation.
682f04e61c58ce852a6c2f7c
Leon Lin
claimed_verified
2025-05-26T08:18:10.145Z
64dedd86288ed5cc73fcd194
/avatars/758ddf534e9efd06463ab3d5e807dfbb.svg
Lin Liangtao
user
LeonTao
682f04e61c58ce852a6c2f7d
Jun Zheng
claimed_verified
2025-05-26T08:18:12.723Z
6381847a471a4550ff298c63
https://cdn-avatars.huggingface.co/v1/production/uploads/6381847a471a4550ff298c63/RTKepvX67R6pLiiUidpUO.png
Jun
user
zxbsmk
682f04e61c58ce852a6c2f7e
Haidong Wang
WebNovelBench: Placing LLM Novelists on the Web Novel Distribution
Robustly evaluating the long-form storytelling capabilities of Large Language
Models (LLMs) remains a significant challenge, as existing benchmarks often
lack the necessary scale, diversity, or objective measures. To address this, we
introduce WebNovelBench, a novel benchmark specifically designed for evaluating
long-form novel generation. WebNovelBench leverages a large-scale dataset of
over 4,000 Chinese web novels, framing evaluation as a synopsis-to-story
generation task. We propose a multi-faceted framework encompassing eight
narrative quality dimensions, assessed automatically via an LLM-as-Judge
approach. Scores are aggregated using Principal Component Analysis and mapped
to a percentile rank against human-authored works. Our experiments demonstrate
that WebNovelBench effectively differentiates between human-written
masterpieces, popular web novels, and LLM-generated content. We provide a
comprehensive analysis of 24 state-of-the-art LLMs, ranking their storytelling
abilities and offering insights for future development. This benchmark provides
a scalable, replicable, and data-driven methodology for assessing and advancing
LLM-driven narrative generation.
WebNovelBench evaluates LLM storytelling capabilities using a large-scale dataset of Chinese web novels, assessing narrative quality across eight dimensions through an LLM-as-Judge framework.
Large Language Models LLM LLM-as-Judge WebNovelBench synopsis-to-story generation narrative quality dimensions Principal Component Analysis percentile rank human-authored works state-of-the-art LLMs narrative generation