A Controlled Study on Long Context Extension and Generalization in LLMs
Broad textual understanding and in-context learning require language models
that utilize full document contexts. Due to the implementation challenges
associated with directly training long-context models, many methods have been
proposed for extending models to handle long contexts. However, owing to
differences in data and model classes, it has been challenging to compare these
approaches, leading to uncertainty as to how to evaluate long-context
performance and whether it differs from standard evaluation. We implement a
controlled protocol for extension methods with a standardized evaluation,
utilizing consistent base models and extension data. Our study yields several
insights into long-context behavior. First, we reaffirm the critical role of
perplexity as a general-purpose performance indicator even in longer-context
tasks. Second, we find that current approximate attention methods
systematically underperform across long-context tasks. Finally, we confirm that
exact fine-tuning based methods are generally effective within the range of
their extension, whereas extrapolation remains challenging. All codebases,
models, and checkpoints will be made available open-source, promoting
transparency and facilitating further research in this critical area of AI
development.
66eba4702f524b05fa0b0fd4
Yi Lu
66eba4702f524b05fa0b0fd5
Jing Nathan Yan
66eba4702f524b05fa0b0fd6
Songlin Yang
admin_assigned
2024-09-19T08:57:13.775Z
65810ad10d738799ecbce2de
https://cdn-avatars.huggingface.co/v1/production/uploads/65810ad10d738799ecbce2de/e0u0Eo-4Wiahw_-cCHvIx.jpeg
Songlin Yang
user
sonta7
66eba4702f524b05fa0b0fd7
Justin T. Chiu
admin_assigned
2024-09-19T08:59:41.988Z
629f763f540c0da640ae7e70
/avatars/1e8005eca079748365e7243a5d100fe9.svg
Justin T Chiu
user
justintchiu
66eba4702f524b05fa0b0fd8
Siyu Ren
66eba4702f524b05fa0b0fd9
Fei Yuan
claimed_verified
2024-09-24T10:20:06.493Z
65fed45b08d35929362dd651
https://cdn-avatars.huggingface.co/v1/production/uploads/65fed45b08d35929362dd651/KLMxsyRN6_HhCZP1iDw6K.png
FeiYuan
user
FeYuan
66eba4702f524b05fa0b0fda
Wenting Zhao
claimed_verified
2024-09-20T07:46:47.166Z
63ffdbaab09f82a81a222c27
/avatars/3aca53f6555f4548489844902fe4a80a.svg
Wenting Zhao
user
wentingzhao
66eba4702f524b05fa0b0fdb
Zhiyong Wu
admin_assigned
2024-09-19T08:59:56.257Z
6280e830e99dccaac4bbfde5
/avatars/9242b8d2826ce2f79af9bb794bba2b61.svg
Zhiyong Wu
user
zy001
66eba4702f524b05fa0b0fdc
Alexander M. Rush
claimed_verified
2024-09-20T07:47:26.315Z
5e694f697127cb58dde42fe1
https://cdn-avatars.huggingface.co/v1/production/uploads/5dd96eb166059660ed1ee413/J21QWWtFOKqhJTF5qIMlD.jpeg
Sasha Rush
user
srush
A Controlled Study on Long Context Extension and Generalization in LLMs
Broad textual understanding and in-context learning require language models
that utilize full document contexts. Due to the implementation challenges
associated with directly training long-context models, many methods have been
proposed for extending models to handle long contexts. However, owing to
differences in data and model classes, it has been challenging to compare these
approaches, leading to uncertainty as to how to evaluate long-context
performance and whether it differs from standard evaluation. We implement a
controlled protocol for extension methods with a standardized evaluation,
utilizing consistent base models and extension data. Our study yields several
insights into long-context behavior. First, we reaffirm the critical role of
perplexity as a general-purpose performance indicator even in longer-context
tasks. Second, we find that current approximate attention methods
systematically underperform across long-context tasks. Finally, we confirm that
exact fine-tuning based methods are generally effective within the range of
their extension, whereas extrapolation remains challenging. All codebases,
models, and checkpoints will be made available open-source, promoting
transparency and facilitating further research in this critical area of AI
development.
A controlled evaluation protocol for long-context models reveals that exact fine-tuning methods are effective, while approximate attention methods underperform, and extrapolation remains challenging.
perplexity approximate attention fine-tuning