title: Evaluating Language Model Context Windows: A "Working Memory" Test and
    Inference-time Correction
summary: Large language models are prominently used in real-world applications, often
  tasked with reasoning over large volumes of documents. An exciting development
  in this space is models boasting extended context capabilities, with some
  accommodating over 2 million tokens. Such long context model capabilities
  remain uncertain in production systems, motivating the need to benchmark their
  performance on real world use cases. We address this challenge by proposing
  SWiM, an evaluation framework that addresses the limitations of standard tests.
  Testing the framework on eight long context models, we find that even strong
  models such as GPT-4 and Claude 3 Opus degrade in performance when information
  is present in the middle of the context window (lost-in-the-middle effect).
  Next, in addition to our benchmark, we propose medoid voting, a simple, but
  effective training-free approach that helps alleviate this effect, by
  generating responses a few times, each time randomly permuting documents in the
  context, and selecting the medoid answer. We evaluate medoid voting on single
  document QA tasks, achieving up to a 24% lift in accuracy.
paper_authors._id: 668d67ba63d1bc005bd5e328
paper_authors.name: Amanda Dsouza
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-09T21:09:03.551Z
paper_authors._id: 654d53ac6db202b835339be9
paper_authors.avatarUrl: /avatars/01d3046e63b3867d5faa6d64cdef09d7.svg
paper_authors.fullname: Amanda Dsouza
paper_authors.type: user
paper_authors.user: andsouzasnorkelai
paper_authors._id: 668d67ba63d1bc005bd5e329
paper_authors.name: Christopher Glaze
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-09T21:09:01.496Z
paper_authors._id: 644985b8eb7db8f70fb72fab
paper_authors.avatarUrl: /avatars/f9861800bd9ad0645af2da9d0db4bcbc.svg
paper_authors.fullname: Christopher Glaze
paper_authors.type: user
paper_authors.user: cmglaze
paper_authors._id: 668d67ba63d1bc005bd5e32a
paper_authors.name: Changho Shin
paper_authors._id: 668d67ba63d1bc005bd5e32b
paper_authors.name: Frederic Sala
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-10T08:19:58.464Z
paper_authors._id: 63044153fc783bfc74461fa9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63044153fc783bfc74461fa9/O5Jz6VPvfVHFyaXH_XqoN.jpeg
paper_authors.fullname: Frederic Sala
paper_authors.type: user
paper_authors.user: fredsala
paper_title: Evaluating Language Model Context Windows: A "Working Memory" Test and
    Inference-time Correction
paper_summary: Large language models are prominently used in real-world applications, often
  tasked with reasoning over large volumes of documents. An exciting development
  in this space is models boasting extended context capabilities, with some
  accommodating over 2 million tokens. Such long context model capabilities
  remain uncertain in production systems, motivating the need to benchmark their
  performance on real world use cases. We address this challenge by proposing
  SWiM, an evaluation framework that addresses the limitations of standard tests.
  Testing the framework on eight long context models, we find that even strong
  models such as GPT-4 and Claude 3 Opus degrade in performance when information
  is present in the middle of the context window (lost-in-the-middle effect).
  Next, in addition to our benchmark, we propose medoid voting, a simple, but
  effective training-free approach that helps alleviate this effect, by
  generating responses a few times, each time randomly permuting documents in the
  context, and selecting the medoid answer. We evaluate medoid voting on single
  document QA tasks, achieving up to a 24% lift in accuracy.
paper_ai_keywords: extended context capabilities context capabilities context window SWiM evaluation framework GPT-4 Claude 3 Opus lost-in-the-middle effect medoid voting training-free approach single document QA tasks