title: Make Your LLM Fully Utilize the Context
summary: While many contemporary large language models (LLMs) can process lengthy
  input, they still struggle to fully utilize information within the long
  context, known as the lost-in-the-middle challenge. We hypothesize that it
  stems from insufficient explicit supervision during the long-context training,
  which fails to emphasize that any position in a long context can hold crucial
  information. Based on this intuition, our study presents information-intensive
  (IN2) training, a purely data-driven solution to overcome lost-in-the-middle.
  Specifically, IN2 training leverages a synthesized long-context question-answer
  dataset, where the answer requires (1) fine-grained information awareness on a
  short segment (~128 tokens) within a synthesized long context (4K-32K tokens),
  and (2) the integration and reasoning of information from two or more short
  segments. Through applying this information-intensive training on Mistral-7B,
  we present FILM-7B (FILl-in-the-Middle). To thoroughly assess the ability of
  FILM-7B for utilizing long contexts, we design three probing tasks that
  encompass various context styles (document, code, and structured-data context)
  and information retrieval patterns (forward, backward, and bi-directional
  retrieval). The probing results demonstrate that FILM-7B can robustly retrieve
  information from different positions in its 32K context window. Beyond these
  probing tasks, FILM-7B significantly improves the performance on real-world
  long-context tasks (e.g., 23.5->26.9 F1 score on NarrativeQA), while
  maintaining a comparable performance on short-context tasks (e.g., 59.3->59.2
  accuracy on MMLU). Github Link: https://github.com/microsoft/FILM.
paper_authors._id: 662b1d65329e3263432600f7
paper_authors.name: Shengnan An
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-26T09:04:00.237Z
paper_authors._id: 64db5f5dd68a6ddcc7bd89e9
paper_authors.avatarUrl: /avatars/69375ec915927b855813df8a6d486837.svg
paper_authors.fullname: Shengnan An
paper_authors.type: user
paper_authors.user: ShengnanAn
paper_authors._id: 662b1d65329e3263432600f8
paper_authors.name: Zexiong Ma
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-30T08:09:57.458Z
paper_authors._id: 654da66fb36f85a025bc24b6
paper_authors.avatarUrl: /avatars/e5542856ab4bf1845e8f546b5f17cd99.svg
paper_authors.fullname: Zexiong Ma
paper_authors.type: user
paper_authors.user: mizersy
paper_authors._id: 662b1d65329e3263432600f9
paper_authors.name: Zeqi Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-26T09:08:12.357Z
paper_authors._id: 642a862525442667e9ba5e78
paper_authors.avatarUrl: /avatars/f15bba13cb4096f6b3f3508ac2c56e5d.svg
paper_authors.fullname: Zeqi Lin
paper_authors.type: user
paper_authors.user: linzeqipku
paper_authors._id: 662b1d65329e3263432600fa
paper_authors.name: Nanning Zheng
paper_authors._id: 662b1d65329e3263432600fb
paper_authors.name: Jian-Guang Lou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-26T09:08:28.504Z
paper_authors._id: 6554338ba4de813482e91ad5
paper_authors.avatarUrl: /avatars/bee879fa4a818fd6c070ee001769ce05.svg
paper_authors.fullname: Jian-Guang Lou
paper_authors.type: user
paper_authors.user: substill
paper_title: Make Your LLM Fully Utilize the Context
paper_summary: While many contemporary large language models (LLMs) can process lengthy
  input, they still struggle to fully utilize information within the long
  context, known as the lost-in-the-middle challenge. We hypothesize that it
  stems from insufficient explicit supervision during the long-context training,
  which fails to emphasize that any position in a long context can hold crucial
  information. Based on this intuition, our study presents information-intensive
  (IN2) training, a purely data-driven solution to overcome lost-in-the-middle.
  Specifically, IN2 training leverages a synthesized long-context question-answer
  dataset, where the answer requires (1) fine-grained information awareness on a
  short segment (~128 tokens) within a synthesized long context (4K-32K tokens),
  and (2) the integration and reasoning of information from two or more short
  segments. Through applying this information-intensive training on Mistral-7B,
  we present FILM-7B (FILl-in-the-Middle). To thoroughly assess the ability of
  FILM-7B for utilizing long contexts, we design three probing tasks that
  encompass various context styles (document, code, and structured-data context)
  and information retrieval patterns (forward, backward, and bi-directional
  retrieval). The probing results demonstrate that FILM-7B can robustly retrieve
  information from different positions in its 32K context window. Beyond these
  probing tasks, FILM-7B significantly improves the performance on real-world
  long-context tasks (e.g., 23.5->26.9 F1 score on NarrativeQA), while
  maintaining a comparable performance on short-context tasks (e.g., 59.3->59.2
  accuracy on MMLU). Github Link: https://github.com/microsoft/FILM.
paper_ai_summary: FILM-7B enhances long-context processing by using information-intensive training with synthesized datasets, improving retrieval and performance on real-world tasks without compromising short-context performance.
paper_ai_keywords: information-intensive training long-context training synthesized long-context question-answer dataset fine-grained information awareness FILM-7B probing tasks document context code context structured-data context forward retrieval backward retrieval bi-directional retrieval NarrativeQA MMLU