title: FocusLLM: Scaling LLM's Context by Parallel Decoding
summary: Empowering LLMs with the ability to utilize useful information from a long
  context is crucial for many downstream applications. However, achieving long
  context lengths with the conventional transformer architecture requires
  substantial training and inference resources. In this paper, we present
  FocusLLM, a framework designed to extend the context length of any decoder-only
  LLM, enabling the model to focus on relevant information from very long
  sequences. FocusLLM processes long text inputs by dividing them into chunks
  based on the model's original context length to alleviate the issue of
  attention distraction. Then, it appends the local context to each chunk as a
  prompt to extract essential information from each chunk based on a novel
  parallel decoding mechanism, and ultimately integrates the extracted
  information into the local context. FocusLLM stands out for great training
  efficiency and versatility: trained with an 8K input length with much less
  training cost than previous methods, FocusLLM exhibits superior performance
  across downstream long-context tasks and maintains strong language modeling
  ability when handling extensive long texts, even up to 400K tokens. Our code is
  available at https://github.com/leezythu/FocusLLM.
paper_authors._id: 66c6a8c20f390b546ae64da2
paper_authors.name: Zhenyu Li
paper_authors._id: 66c6a8c20f390b546ae64da3
paper_authors.name: Yike Zhang
paper_authors._id: 66c6a8c20f390b546ae64da4
paper_authors.name: Tengyu Pan
paper_authors._id: 66c6a8c20f390b546ae64da5
paper_authors.name: Yutao Sun
paper_authors._id: 66c6a8c20f390b546ae64da6
paper_authors.name: Zhichao Duan
paper_authors._id: 66c6a8c20f390b546ae64da7
paper_authors.name: Junjie Fang
paper_authors._id: 66c6a8c20f390b546ae64da8
paper_authors.name: Rong Han
paper_authors._id: 66c6a8c20f390b546ae64da9
paper_authors.name: Zixuan Wang
paper_authors._id: 66c6a8c20f390b546ae64daa
paper_authors.name: Jianyong Wang
paper_title: FocusLLM: Scaling LLM's Context by Parallel Decoding
paper_summary: Empowering LLMs with the ability to utilize useful information from a long
  context is crucial for many downstream applications. However, achieving long
  context lengths with the conventional transformer architecture requires
  substantial training and inference resources. In this paper, we present
  FocusLLM, a framework designed to extend the context length of any decoder-only
  LLM, enabling the model to focus on relevant information from very long
  sequences. FocusLLM processes long text inputs by dividing them into chunks
  based on the model's original context length to alleviate the issue of
  attention distraction. Then, it appends the local context to each chunk as a
  prompt to extract essential information from each chunk based on a novel
  parallel decoding mechanism, and ultimately integrates the extracted
  information into the local context. FocusLLM stands out for great training
  efficiency and versatility: trained with an 8K input length with much less
  training cost than previous methods, FocusLLM exhibits superior performance
  across downstream long-context tasks and maintains strong language modeling
  ability when handling extensive long texts, even up to 400K tokens. Our code is
  available at https://github.com/leezythu/FocusLLM.
paper_ai_keywords: transformer architecture decoder-only LLM FocusLLM attention distraction parallel decoding mechanism long-context tasks language modeling