title: L-CiteEval: Do Long-Context Models Truly Leverage Context for
    Responding?
summary: Long-context models (LCMs) have made remarkable strides in recent years,
  offering users great convenience for handling tasks that involve long context,
  such as document summarization. As the community increasingly prioritizes the
  faithfulness of generated results, merely ensuring the accuracy of LCM outputs
  is insufficient, as it is quite challenging for humans to verify the results
  from the extremely lengthy context. Yet, although some efforts have been made
  to assess whether LCMs respond truly based on the context, these works either
  are limited to specific tasks or heavily rely on external evaluation resources
  like GPT-4.In this work, we introduce L-CiteEval, a comprehensive multi-task
  benchmark for long-context understanding with citations, aiming to evaluate
  both the understanding capability and faithfulness of LCMs. L-CiteEval covers
  11 tasks from diverse domains, spanning context lengths from 8K to 48K, and
  provides a fully automated evaluation suite. Through testing with 11
  cutting-edge closed-source and open-source LCMs, we find that although these
  models show minor differences in their generated results, open-source models
  substantially trail behind their closed-source counterparts in terms of
  citation accuracy and recall. This suggests that current open-source LCMs are
  prone to responding based on their inherent knowledge rather than the given
  context, posing a significant risk to the user experience in practical
  applications. We also evaluate the RAG approach and observe that RAG can
  significantly improve the faithfulness of LCMs, albeit with a slight decrease
  in the generation quality. Furthermore, we discover a correlation between the
  attention mechanisms of LCMs and the citation generation process.
paper_authors._id: 66ff774b031160f2965e06b1
paper_authors.name: Zecheng Tang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-04T09:02:57.554Z
paper_authors._id: 64096ef79e9f790c905b846d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64096ef79e9f790c905b846d/hVzw656lXdzbCxTtnheud.jpeg
paper_authors.fullname: Zecheng Tang
paper_authors.type: user
paper_authors.user: ZetangForward
paper_authors._id: 66ff774b031160f2965e06b2
paper_authors.name: Keyan Zhou
paper_authors._id: 66ff774b031160f2965e06b3
paper_authors.name: Juntao Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-04T20:11:25.296Z
paper_authors._id: 60d1d7f94672e8c13e7c2bce
paper_authors.avatarUrl: /avatars/be3a3990492ef1c5f3ff33cc043ae166.svg
paper_authors.fullname: Juntao Li
paper_authors.type: user
paper_authors.user: ljtsuda
paper_authors._id: 66ff774b031160f2965e06b4
paper_authors.name: Baibei Ji
paper_authors._id: 66ff774b031160f2965e06b5
paper_authors.name: Jianye Hou
paper_authors._id: 66ff774b031160f2965e06b6
paper_authors.name: Min Zhang
paper_title: L-CiteEval: Do Long-Context Models Truly Leverage Context for
    Responding?
paper_summary: Long-context models (LCMs) have made remarkable strides in recent years,
  offering users great convenience for handling tasks that involve long context,
  such as document summarization. As the community increasingly prioritizes the
  faithfulness of generated results, merely ensuring the accuracy of LCM outputs
  is insufficient, as it is quite challenging for humans to verify the results
  from the extremely lengthy context. Yet, although some efforts have been made
  to assess whether LCMs respond truly based on the context, these works either
  are limited to specific tasks or heavily rely on external evaluation resources
  like GPT-4.In this work, we introduce L-CiteEval, a comprehensive multi-task
  benchmark for long-context understanding with citations, aiming to evaluate
  both the understanding capability and faithfulness of LCMs. L-CiteEval covers
  11 tasks from diverse domains, spanning context lengths from 8K to 48K, and
  provides a fully automated evaluation suite. Through testing with 11
  cutting-edge closed-source and open-source LCMs, we find that although these
  models show minor differences in their generated results, open-source models
  substantially trail behind their closed-source counterparts in terms of
  citation accuracy and recall. This suggests that current open-source LCMs are
  prone to responding based on their inherent knowledge rather than the given
  context, posing a significant risk to the user experience in practical
  applications. We also evaluate the RAG approach and observe that RAG can
  significantly improve the faithfulness of LCMs, albeit with a slight decrease
  in the generation quality. Furthermore, we discover a correlation between the
  attention mechanisms of LCMs and the citation generation process.
paper_ai_keywords: Long-context models (LCMs) document summarization faithfulness of generated results denoising autoencoders L-CiteEval multi-task benchmark long-context understanding citations context lengths fully automated evaluation suite citation accuracy recall RAG (Retriever-Analyzer-Generator) approach attention mechanisms citation generation process