title: A Controlled Study on Long Context Extension and Generalization in LLMs
summary: Broad textual understanding and in-context learning require language models
  that utilize full document contexts. Due to the implementation challenges
  associated with directly training long-context models, many methods have been
  proposed for extending models to handle long contexts. However, owing to
  differences in data and model classes, it has been challenging to compare these
  approaches, leading to uncertainty as to how to evaluate long-context
  performance and whether it differs from standard evaluation. We implement a
  controlled protocol for extension methods with a standardized evaluation,
  utilizing consistent base models and extension data. Our study yields several
  insights into long-context behavior. First, we reaffirm the critical role of
  perplexity as a general-purpose performance indicator even in longer-context
  tasks. Second, we find that current approximate attention methods
  systematically underperform across long-context tasks. Finally, we confirm that
  exact fine-tuning based methods are generally effective within the range of
  their extension, whereas extrapolation remains challenging. All codebases,
  models, and checkpoints will be made available open-source, promoting
  transparency and facilitating further research in this critical area of AI
  development.
paper_authors._id: 66eba4702f524b05fa0b0fd4
paper_authors.name: Yi Lu
paper_authors._id: 66eba4702f524b05fa0b0fd5
paper_authors.name: Jing Nathan Yan
paper_authors._id: 66eba4702f524b05fa0b0fd6
paper_authors.name: Songlin Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-19T08:57:13.775Z
paper_authors._id: 65810ad10d738799ecbce2de
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65810ad10d738799ecbce2de/e0u0Eo-4Wiahw_-cCHvIx.jpeg
paper_authors.fullname: Songlin Yang
paper_authors.type: user
paper_authors.user: sonta7
paper_authors._id: 66eba4702f524b05fa0b0fd7
paper_authors.name: Justin T. Chiu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-19T08:59:41.988Z
paper_authors._id: 629f763f540c0da640ae7e70
paper_authors.avatarUrl: /avatars/1e8005eca079748365e7243a5d100fe9.svg
paper_authors.fullname: Justin T Chiu
paper_authors.type: user
paper_authors.user: justintchiu
paper_authors._id: 66eba4702f524b05fa0b0fd8
paper_authors.name: Siyu Ren
paper_authors._id: 66eba4702f524b05fa0b0fd9
paper_authors.name: Fei Yuan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-24T10:20:06.493Z
paper_authors._id: 65fed45b08d35929362dd651
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65fed45b08d35929362dd651/KLMxsyRN6_HhCZP1iDw6K.png
paper_authors.fullname: FeiYuan
paper_authors.type: user
paper_authors.user: FeYuan
paper_authors._id: 66eba4702f524b05fa0b0fda
paper_authors.name: Wenting Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-20T07:46:47.166Z
paper_authors._id: 63ffdbaab09f82a81a222c27
paper_authors.avatarUrl: /avatars/3aca53f6555f4548489844902fe4a80a.svg
paper_authors.fullname: Wenting Zhao
paper_authors.type: user
paper_authors.user: wentingzhao
paper_authors._id: 66eba4702f524b05fa0b0fdb
paper_authors.name: Zhiyong Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-19T08:59:56.257Z
paper_authors._id: 6280e830e99dccaac4bbfde5
paper_authors.avatarUrl: /avatars/9242b8d2826ce2f79af9bb794bba2b61.svg
paper_authors.fullname: Zhiyong Wu
paper_authors.type: user
paper_authors.user: zy001
paper_authors._id: 66eba4702f524b05fa0b0fdc
paper_authors.name: Alexander M. Rush
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-20T07:47:26.315Z
paper_authors._id: 5e694f697127cb58dde42fe1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5dd96eb166059660ed1ee413/J21QWWtFOKqhJTF5qIMlD.jpeg
paper_authors.fullname: Sasha Rush
paper_authors.type: user
paper_authors.user: srush
paper_title: A Controlled Study on Long Context Extension and Generalization in LLMs
paper_summary: Broad textual understanding and in-context learning require language models
  that utilize full document contexts. Due to the implementation challenges
  associated with directly training long-context models, many methods have been
  proposed for extending models to handle long contexts. However, owing to
  differences in data and model classes, it has been challenging to compare these
  approaches, leading to uncertainty as to how to evaluate long-context
  performance and whether it differs from standard evaluation. We implement a
  controlled protocol for extension methods with a standardized evaluation,
  utilizing consistent base models and extension data. Our study yields several
  insights into long-context behavior. First, we reaffirm the critical role of
  perplexity as a general-purpose performance indicator even in longer-context
  tasks. Second, we find that current approximate attention methods
  systematically underperform across long-context tasks. Finally, we confirm that
  exact fine-tuning based methods are generally effective within the range of
  their extension, whereas extrapolation remains challenging. All codebases,
  models, and checkpoints will be made available open-source, promoting
  transparency and facilitating further research in this critical area of AI
  development.
paper_ai_keywords: approximate attention methods exact fine-tuning based methods