title: WebNovelBench: Placing LLM Novelists on the Web Novel Distribution
summary: Robustly evaluating the long-form storytelling capabilities of Large Language
  Models (LLMs) remains a significant challenge, as existing benchmarks often
  lack the necessary scale, diversity, or objective measures. To address this, we
  introduce WebNovelBench, a novel benchmark specifically designed for evaluating
  long-form novel generation. WebNovelBench leverages a large-scale dataset of
  over 4,000 Chinese web novels, framing evaluation as a synopsis-to-story
  generation task. We propose a multi-faceted framework encompassing eight
  narrative quality dimensions, assessed automatically via an LLM-as-Judge
  approach. Scores are aggregated using Principal Component Analysis and mapped
  to a percentile rank against human-authored works. Our experiments demonstrate
  that WebNovelBench effectively differentiates between human-written
  masterpieces, popular web novels, and LLM-generated content. We provide a
  comprehensive analysis of 24 state-of-the-art LLMs, ranking their storytelling
  abilities and offering insights for future development. This benchmark provides
  a scalable, replicable, and data-driven methodology for assessing and advancing
  LLM-driven narrative generation.
paper_authors._id: 682f04e61c58ce852a6c2f7c
paper_authors.name: Leon Lin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-26T08:18:10.145Z
paper_authors._id: 64dedd86288ed5cc73fcd194
paper_authors.avatarUrl: /avatars/758ddf534e9efd06463ab3d5e807dfbb.svg
paper_authors.fullname: Lin Liangtao
paper_authors.type: user
paper_authors.user: LeonTao
paper_authors._id: 682f04e61c58ce852a6c2f7d
paper_authors.name: Jun Zheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-26T08:18:12.723Z
paper_authors._id: 6381847a471a4550ff298c63
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6381847a471a4550ff298c63/RTKepvX67R6pLiiUidpUO.png
paper_authors.fullname: Jun
paper_authors.type: user
paper_authors.user: zxbsmk
paper_authors._id: 682f04e61c58ce852a6c2f7e
paper_authors.name: Haidong Wang
paper_title: WebNovelBench: Placing LLM Novelists on the Web Novel Distribution
paper_summary: Robustly evaluating the long-form storytelling capabilities of Large Language
  Models (LLMs) remains a significant challenge, as existing benchmarks often
  lack the necessary scale, diversity, or objective measures. To address this, we
  introduce WebNovelBench, a novel benchmark specifically designed for evaluating
  long-form novel generation. WebNovelBench leverages a large-scale dataset of
  over 4,000 Chinese web novels, framing evaluation as a synopsis-to-story
  generation task. We propose a multi-faceted framework encompassing eight
  narrative quality dimensions, assessed automatically via an LLM-as-Judge
  approach. Scores are aggregated using Principal Component Analysis and mapped
  to a percentile rank against human-authored works. Our experiments demonstrate
  that WebNovelBench effectively differentiates between human-written
  masterpieces, popular web novels, and LLM-generated content. We provide a
  comprehensive analysis of 24 state-of-the-art LLMs, ranking their storytelling
  abilities and offering insights for future development. This benchmark provides
  a scalable, replicable, and data-driven methodology for assessing and advancing
  LLM-driven narrative generation.
paper_ai_summary: WebNovelBench evaluates LLM storytelling capabilities using a large-scale dataset of Chinese web novels, assessing narrative quality across eight dimensions through an LLM-as-Judge framework.
paper_ai_keywords: Large Language Models LLM LLM-as-Judge WebNovelBench synopsis-to-story generation narrative quality dimensions Principal Component Analysis percentile rank human-authored works state-of-the-art LLMs narrative generation