title: HelloBench: Evaluating Long Text Generation Capabilities of Large
    Language Models
summary: In recent years, Large Language Models (LLMs) have demonstrated remarkable
  capabilities in various tasks (e.g., long-context understanding), and many
  benchmarks have been proposed. However, we observe that long text generation
  capabilities are not well investigated. Therefore, we introduce the
  Hierarchical Long Text Generation Benchmark (HelloBench), a comprehensive,
  in-the-wild, and open-ended benchmark to evaluate LLMs' performance in
  generating long text. Based on Bloom's Taxonomy, HelloBench categorizes long
  text generation tasks into five subtasks: open-ended QA, summarization, chat,
  text completion, and heuristic text generation. Besides, we propose
  Hierarchical Long Text Evaluation (HelloEval), a human-aligned evaluation
  method that significantly reduces the time and effort required for human
  evaluation while maintaining a high correlation with human evaluation. We have
  conducted extensive experiments across around 30 mainstream LLMs and observed
  that the current LLMs lack long text generation capabilities. Specifically,
  first, regardless of whether the instructions include explicit or implicit
  length constraints, we observe that most LLMs cannot generate text that is
  longer than 4000 words. Second, we observe that while some LLMs can generate
  longer text, many issues exist (e.g., severe repetition and quality
  degradation). Third, to demonstrate the effectiveness of HelloEval, we compare
  HelloEval with traditional metrics (e.g., ROUGE, BLEU, etc.) and LLM-as-a-Judge
  methods, which show that HelloEval has the highest correlation with human
  evaluation. We release our code in https://github.com/Quehry/HelloBench.
paper_authors._id: 66f37983685ee4f1ba09e0f1
paper_authors.name: Haoran Que
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-25T07:32:32.953Z
paper_authors._id: 632176664204950905b64e58
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/632176664204950905b64e58/IzkGIaAWy9285Q8uBncZ6.jpeg
paper_authors.fullname: quehry
paper_authors.type: user
paper_authors.user: quehry
paper_authors._id: 66f37983685ee4f1ba09e0f2
paper_authors.name: Feiyu Duan
paper_authors._id: 66f37983685ee4f1ba09e0f3
paper_authors.name: Liqun He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-25T18:35:59.010Z
paper_authors._id: 64474bd91f308334030d086a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64474bd91f308334030d086a/TkyI2tfOMEYF9Sdja20Of.jpeg
paper_authors.fullname: Liqun He
paper_authors.type: user
paper_authors.user: liqunhe27
paper_authors._id: 66f37983685ee4f1ba09e0f4
paper_authors.name: Yutao Mou
paper_authors._id: 66f37983685ee4f1ba09e0f5
paper_authors.name: Wangchunshu Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-25T18:36:11.181Z
paper_authors._id: 628c8598ef14f971b698107f
paper_authors.avatarUrl: /avatars/3a4ad87e6b5f9e836a1160d869df1447.svg
paper_authors.fullname: Zhou
paper_authors.type: user
paper_authors.user: Wangchunshu
paper_authors._id: 66f37983685ee4f1ba09e0f6
paper_authors.name: Jiaheng Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-25T18:36:17.540Z
paper_authors._id: 65377c30e48353201e6fdda0
paper_authors.avatarUrl: /avatars/a8f803b6f2e598eaee9c52c0d2ddfc16.svg
paper_authors.fullname: Jiaheng Liu
paper_authors.type: user
paper_authors.user: CheeryLJH
paper_authors._id: 66f37983685ee4f1ba09e0f7
paper_authors.name: Wenge Rong
paper_authors._id: 66f37983685ee4f1ba09e0f8
paper_authors.name: Zekun Moore Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-25T18:36:30.673Z
paper_authors._id: 6149a9e95347647e6bb68882
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6149a9e95347647e6bb68882/Jddln1FxScCeVgTSCNBpr.png
paper_authors.fullname: Zekun Moore Wang
paper_authors.type: user
paper_authors.user: ZenMoore
paper_authors._id: 66f37983685ee4f1ba09e0f9
paper_authors.name: Jian Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-14T19:01:14.371Z
paper_authors._id: 64ccb9bfead94891d12aef42
paper_authors.avatarUrl: /avatars/c54809d43d93d3f0766bd2555cecc4e3.svg
paper_authors.fullname: Yang Jian
paper_authors.type: user
paper_authors.user: CSJianYang
paper_authors._id: 66f37983685ee4f1ba09e0fa
paper_authors.name: Ge Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-25T07:32:35.486Z
paper_authors._id: 638efcf4c67af472d316d424
paper_authors.avatarUrl: /avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
paper_authors.fullname: Ge Zhang
paper_authors.type: user
paper_authors.user: zhangysk
paper_authors._id: 66f37983685ee4f1ba09e0fb
paper_authors.name: Junran Peng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-25T18:36:39.841Z
paper_authors._id: 62c11a7360edb2dd7765b80d
paper_authors.avatarUrl: /avatars/92db3258d0ba44daee22952f0644cd93.svg
paper_authors.fullname: Junran Peng
paper_authors.type: user
paper_authors.user: JrPeng
paper_authors._id: 66f37983685ee4f1ba09e0fc
paper_authors.name: Zhaoxiang Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-25T18:36:50.394Z
paper_authors._id: 6350d59989def14ad21e11b3
paper_authors.avatarUrl: /avatars/9a7ae6131e0a89f3461f267ee844b218.svg
paper_authors.fullname: z
paper_authors.type: user
paper_authors.user: smallbert
paper_authors._id: 66f37983685ee4f1ba09e0fd
paper_authors.name: Songyang Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-25T07:32:28.146Z
paper_authors._id: 630716d11801ecc7d2595021
paper_authors.avatarUrl: /avatars/2d36a880ce4a3cf7efc5ff3987dbeaf3.svg
paper_authors.fullname: Songyang Zhang
paper_authors.type: user
paper_authors.user: zsytony
paper_authors._id: 66f37983685ee4f1ba09e0fe
paper_authors.name: Kai Chen
paper_title: HelloBench: Evaluating Long Text Generation Capabilities of Large
    Language Models
paper_summary: In recent years, Large Language Models (LLMs) have demonstrated remarkable
  capabilities in various tasks (e.g., long-context understanding), and many
  benchmarks have been proposed. However, we observe that long text generation
  capabilities are not well investigated. Therefore, we introduce the
  Hierarchical Long Text Generation Benchmark (HelloBench), a comprehensive,
  in-the-wild, and open-ended benchmark to evaluate LLMs' performance in
  generating long text. Based on Bloom's Taxonomy, HelloBench categorizes long
  text generation tasks into five subtasks: open-ended QA, summarization, chat,
  text completion, and heuristic text generation. Besides, we propose
  Hierarchical Long Text Evaluation (HelloEval), a human-aligned evaluation
  method that significantly reduces the time and effort required for human
  evaluation while maintaining a high correlation with human evaluation. We have
  conducted extensive experiments across around 30 mainstream LLMs and observed
  that the current LLMs lack long text generation capabilities. Specifically,
  first, regardless of whether the instructions include explicit or implicit
  length constraints, we observe that most LLMs cannot generate text that is
  longer than 4000 words. Second, we observe that while some LLMs can generate
  longer text, many issues exist (e.g., severe repetition and quality
  degradation). Third, to demonstrate the effectiveness of HelloEval, we compare
  HelloEval with traditional metrics (e.g., ROUGE, BLEU, etc.) and LLM-as-a-Judge
  methods, which show that HelloEval has the highest correlation with human
  evaluation. We release our code in https://github.com/Quehry/HelloBench.
paper_ai_summary: A comprehensive benchmark, HelloBench, and evaluation method, HelloEval, are introduced to assess the long text generation capabilities of Large Language Models, revealing limitations and providing human-aligned evaluations.
paper_ai_keywords: Large Language Models Hierarchical Long Text Generation Benchmark HelloBench Hierarchical Long Text Evaluation HelloEval Bloom's Taxonomy open-ended QA summarization text completion heuristic text generation ROUGE BLEU LLM-as-a-Judge