title: Spinning the Golden Thread: Benchmarking Long-Form Generation in
    Language Models
summary: The abilities of long-context language models (LMs) are often evaluated using
  the "Needle-in-a-Haystack" (NIAH) test, which comprises tasks designed to
  assess a model's ability to identify specific information ("needle") within
  large text sequences ("haystack"). While these benchmarks measure how well
  models understand long-context input sequences, they do not effectively gauge
  the quality of long-form text generation--a critical aspect for applications
  such as design proposals and creative writing. To address this gap, we have
  introduced a new long-form text evaluation benchmark, Spinning the Golden
  Thread (SGT), which tests models' ability to identify specific events within
  generated long text sequences. In this benchmark, we prompt long-context LMs to
  create long-form text that must include particular events or constraints and
  evaluate their ability to incorporate these elements. We evaluated ten
  long-context LMs across four distinct scenarios, three types of prompt
  instructions, and two different generation-length settings (16K and 32K).
  Although these models perform well on NIAH benchmarks, none demonstrated
  satisfactory performance on the Spinning the Golden Thread, raising concerns
  about their ability to generate coherent long-form text that follows
  instructions. Additionally, as the length of the generated text increases, all
  models exhibit a significant drop in performance.
paper_authors._id: 66df11aeab7f91288b88cc4e
paper_authors.name: Yuhao Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-09T19:34:59.092Z
paper_authors._id: 63369da91ba5d5ece24118a4
paper_authors.avatarUrl: /avatars/67889e1ecadb04100a77bc8b5284c6fd.svg
paper_authors.fullname: wuyuhao
paper_authors.type: user
paper_authors.user: mozhu
paper_authors._id: 66df11aeab7f91288b88cc4f
paper_authors.name: Ming Shan Hee
paper_authors._id: 66df11aeab7f91288b88cc50
paper_authors.name: Zhiqing Hu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-09T19:35:01.146Z
paper_authors._id: 637f228152229c63921119c3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/637f228152229c63921119c3/acwXorra1r9_7i3KlBFjS.jpeg
paper_authors.fullname: Zhiqiang Hu
paper_authors.type: user
paper_authors.user: Zhiqiang007
paper_authors._id: 66df11aeab7f91288b88cc51
paper_authors.name: Roy Ka-Wei Lee
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-10T07:12:04.564Z
paper_authors._id: 64b94cc0e3d41dbd6974ae45
paper_authors.avatarUrl: /avatars/5edb9d9465addfceccef04c4465a34e6.svg
paper_authors.fullname: Roy Ka-Wei lee
paper_authors.type: user
paper_authors.user: sroylee
paper_title: Spinning the Golden Thread: Benchmarking Long-Form Generation in
    Language Models
paper_summary: The abilities of long-context language models (LMs) are often evaluated using
  the "Needle-in-a-Haystack" (NIAH) test, which comprises tasks designed to
  assess a model's ability to identify specific information ("needle") within
  large text sequences ("haystack"). While these benchmarks measure how well
  models understand long-context input sequences, they do not effectively gauge
  the quality of long-form text generation--a critical aspect for applications
  such as design proposals and creative writing. To address this gap, we have
  introduced a new long-form text evaluation benchmark, Spinning the Golden
  Thread (SGT), which tests models' ability to identify specific events within
  generated long text sequences. In this benchmark, we prompt long-context LMs to
  create long-form text that must include particular events or constraints and
  evaluate their ability to incorporate these elements. We evaluated ten
  long-context LMs across four distinct scenarios, three types of prompt
  instructions, and two different generation-length settings (16K and 32K).
  Although these models perform well on NIAH benchmarks, none demonstrated
  satisfactory performance on the Spinning the Golden Thread, raising concerns
  about their ability to generate coherent long-form text that follows
  instructions. Additionally, as the length of the generated text increases, all
  models exhibit a significant drop in performance.
paper_ai_keywords: long-context language models (LMs) Needle-in-a-Haystack (NIAH) test long-form text generation Spinning the Golden Thread (SGT) prompt instructions generation-length settings