title: LongCite: Enabling LLMs to Generate Fine-grained Citations in
    Long-context QA
summary: Though current long-context large language models (LLMs) have demonstrated
  impressive capacities in answering user questions based on extensive text, the
  lack of citations in their responses makes user verification difficult, leading
  to concerns about their trustworthiness due to their potential hallucinations.
  In this work, we aim to enable long-context LLMs to generate responses with
  fine-grained sentence-level citations, improving their faithfulness and
  verifiability. We first introduce LongBench-Cite, an automated benchmark for
  assessing current LLMs' performance in Long-Context Question Answering with
  Citations (LQAC), revealing considerable room for improvement. To this end, we
  propose CoF (Coarse to Fine), a novel pipeline that utilizes off-the-shelf LLMs
  to automatically generate long-context QA instances with precise sentence-level
  citations, and leverage this pipeline to construct LongCite-45k, a large-scale
  SFT dataset for LQAC. Finally, we train LongCite-8B and LongCite-9B using the
  LongCite-45k dataset, successfully enabling their generation of accurate
  responses and fine-grained sentence-level citations in a single output. The
  evaluation results on LongBench-Cite show that our trained models achieve
  state-of-the-art citation quality, surpassing advanced proprietary models
  including GPT-4o.
paper_authors._id: 66d90af260730595a869726c
paper_authors.name: jiajie Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-05T07:24:01.604Z
paper_authors._id: 66cdd285c51a915bd5f2d017
paper_authors.avatarUrl: /avatars/14e5794307e4672b1b51d26b31227e0f.svg
paper_authors.fullname: Jiajie Zhang
paper_authors.type: user
paper_authors.user: NeoZ123
paper_authors._id: 66d90af260730595a869726d
paper_authors.name: Yushi Bai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-05T07:35:53.785Z
paper_authors._id: 64ed568ccf6118a9379a61b8
paper_authors.avatarUrl: /avatars/6d040cbcb4a9b624cbe64c9d01cd5c88.svg
paper_authors.fullname: Yushi Bai
paper_authors.type: user
paper_authors.user: bys0318
paper_authors._id: 66d90af260730595a869726e
paper_authors.name: Xin Lv
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-05T07:23:59.827Z
paper_authors._id: 648c64829c935db2b527a764
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/648c64829c935db2b527a764/AnTu1dvpac3hUxLQo8A8K.jpeg
paper_authors.fullname: Xin Lv
paper_authors.type: user
paper_authors.user: davidlvxin
paper_authors._id: 66d90af260730595a869726f
paper_authors.name: Wanjun Gu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-05T07:35:59.620Z
paper_authors._id: 632d2e46fadf3a596468a74c
paper_authors.avatarUrl: /avatars/8314829e9291eee3b844d08e44e8f0ee.svg
paper_authors.fullname: Wanjun Gu
paper_authors.type: user
paper_authors.user: Broccolito
paper_authors._id: 66d90af260730595a8697270
paper_authors.name: Danqing Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-05T07:36:05.613Z
paper_authors._id: 64ad519f714bad736d49703c
paper_authors.avatarUrl: /avatars/bca409d77ec8ff39abb1f6baaa1d0281.svg
paper_authors.fullname: Danqing Liu
paper_authors.type: user
paper_authors.user: LZDQ
paper_authors._id: 66d90af260730595a8697271
paper_authors.name: Minhao Zou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-05T07:36:11.662Z
paper_authors._id: 65e57d114230174d546cd5ee
paper_authors.avatarUrl: /avatars/42fb7c3bcb5bfda24d8f392fb36ede34.svg
paper_authors.fullname: minhao zou
paper_authors.type: user
paper_authors.user: zoumh21
paper_authors._id: 66d90af260730595a8697272
paper_authors.name: Shulin Cao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-05T07:36:22.153Z
paper_authors._id: 62c924a6334a6ee11c2e8dfa
paper_authors.avatarUrl: /avatars/3dbc37af162b94d68cb83665ac4528c3.svg
paper_authors.fullname: ShulinCao
paper_authors.type: user
paper_authors.user: caoshulin
paper_authors._id: 66d90af260730595a8697273
paper_authors.name: Lei Hou
paper_authors._id: 66d90af260730595a8697274
paper_authors.name: Yuxiao Dong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-05T07:36:33.097Z
paper_authors._id: 640e73bdfdeaae1390857b62
paper_authors.avatarUrl: /avatars/cd6779e30f716002a7838ed93d5c0754.svg
paper_authors.fullname: Yuxiao Dong
paper_authors.type: user
paper_authors.user: yuxiaod
paper_authors._id: 66d90af260730595a8697275
paper_authors.name: Ling Feng
paper_authors._id: 66d90af260730595a8697276
paper_authors.name: Juanzi Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-05T07:36:40.318Z
paper_authors._id: 65df8cbc2705d9672f55d1aa
paper_authors.avatarUrl: /avatars/63e46f15bb76bd9d4508fd0f54f39829.svg
paper_authors.fullname: Juanzi Li
paper_authors.type: user
paper_authors.user: juanli
paper_title: LongCite: Enabling LLMs to Generate Fine-grained Citations in
    Long-context QA
paper_summary: Though current long-context large language models (LLMs) have demonstrated
  impressive capacities in answering user questions based on extensive text, the
  lack of citations in their responses makes user verification difficult, leading
  to concerns about their trustworthiness due to their potential hallucinations.
  In this work, we aim to enable long-context LLMs to generate responses with
  fine-grained sentence-level citations, improving their faithfulness and
  verifiability. We first introduce LongBench-Cite, an automated benchmark for
  assessing current LLMs' performance in Long-Context Question Answering with
  Citations (LQAC), revealing considerable room for improvement. To this end, we
  propose CoF (Coarse to Fine), a novel pipeline that utilizes off-the-shelf LLMs
  to automatically generate long-context QA instances with precise sentence-level
  citations, and leverage this pipeline to construct LongCite-45k, a large-scale
  SFT dataset for LQAC. Finally, we train LongCite-8B and LongCite-9B using the
  LongCite-45k dataset, successfully enabling their generation of accurate
  responses and fine-grained sentence-level citations in a single output. The
  evaluation results on LongBench-Cite show that our trained models achieve
  state-of-the-art citation quality, surpassing advanced proprietary models
  including GPT-4o.
paper_ai_keywords: long-context LLMs citations LongBench-Cite long-context Question Answering with Citations (LQAC) denoising autoencoders synthesis pixel space latent space pretrained autoencoders cross-attention bounding boxes convolutional manner latent diffusion models (LDMs) image inpainting unconditional image generation semantic scene synthesis super-resolution computational requirements long-context LLMs citations LongBench-Cite long-context Question Answering with Citations (LQAC) CoF (Coarse to Fine) SFT dataset LongCite-45k LongCite-8B LongCite-9B sentence-level citations state-of-the-art citation quality long-context