RULER: What's the Real Context Size of Your Long-Context Language
  Models?
The needle-in-a-haystack (NIAH) test, which examines the ability to retrieve
a piece of information (the "needle") from long distractor texts (the
"haystack"), has been widely adopted to evaluate long-context language models
(LMs). However, this simple retrieval-based test is indicative of only a
superficial form of long-context understanding. To provide a more comprehensive
evaluation of long-context LMs, we create a new synthetic benchmark RULER with
flexible configurations for customized sequence length and task complexity.
RULER expands upon the vanilla NIAH test to encompass variations with diverse
types and quantities of needles. Moreover, RULER introduces new task categories
multi-hop tracing and aggregation to test behaviors beyond searching from
context. We evaluate ten long-context LMs with 13 representative tasks in
RULER. Despite achieving nearly perfect accuracy in the vanilla NIAH test, all
models exhibit large performance drops as the context length increases. While
these models all claim context sizes of 32K tokens or greater, only four models
(GPT-4, Command-R, Yi-34B, and Mixtral) can maintain satisfactory performance
at the length of 32K. Our analysis of Yi-34B, which supports context length of
200K, reveals large room for improvement as we increase input length and task
complexity. We open source RULER to spur comprehensive evaluation of
long-context LMs.
66173e869c0e16d020e31ef7
Cheng-Ping Hsieh
claimed_verified
2024-04-11T20:47:44.740Z
651c80b534a090d301fa211a
/avatars/ec34ca524616894b62695d025d4fd6d1.svg
Cheng-Ping Hsieh
user
hsiehjackson
66173e869c0e16d020e31ef8
Simeng Sun
claimed_verified
2024-04-12T05:01:59.073Z
62cd08c1ff01ad1c9e509bc9
/avatars/2dc8a0c381962e8c35c6571c406b46ab.svg
Simeng Sun
user
simsun131
66173e869c0e16d020e31ef9
Samuel Kriman
admin_assigned
2024-04-18T12:40:54.419Z
63b884a1aaa8cf17f0975921
/avatars/fc3c807b68d190ea5606ca2c30c3fad6.svg
Samuel Kriman
user
nv-sam
66173e869c0e16d020e31efa
Shantanu Acharya
claimed_verified
2024-04-11T14:54:49.981Z
626c5978f451470f86251117
https://cdn-avatars.huggingface.co/v1/production/uploads/626c5978f451470f86251117/KgtS2aFEMsSF1z79T3sob.jpeg
Shantanu Acharya
user
shantanuacharya
66173e869c0e16d020e31efb
Dima Rekesh
66173e869c0e16d020e31efc
Fei Jia
66173e869c0e16d020e31efd
Boris Ginsburg
RULER: What's the Real Context Size of Your Long-Context Language
  Models?
The needle-in-a-haystack (NIAH) test, which examines the ability to retrieve
a piece of information (the "needle") from long distractor texts (the
"haystack"), has been widely adopted to evaluate long-context language models
(LMs). However, this simple retrieval-based test is indicative of only a
superficial form of long-context understanding. To provide a more comprehensive
evaluation of long-context LMs, we create a new synthetic benchmark RULER with
flexible configurations for customized sequence length and task complexity.
RULER expands upon the vanilla NIAH test to encompass variations with diverse
types and quantities of needles. Moreover, RULER introduces new task categories
multi-hop tracing and aggregation to test behaviors beyond searching from
context. We evaluate ten long-context LMs with 13 representative tasks in
RULER. Despite achieving nearly perfect accuracy in the vanilla NIAH test, all
models exhibit large performance drops as the context length increases. While
these models all claim context sizes of 32K tokens or greater, only four models
(GPT-4, Command-R, Yi-34B, and Mixtral) can maintain satisfactory performance
at the length of 32K. Our analysis of Yi-34B, which supports context length of
200K, reveals large room for improvement as we increase input length and task
complexity. We open source RULER to spur comprehensive evaluation of
long-context LMs.
RULER is a comprehensive benchmark that extends beyond simple retrieval tasks to evaluate long-context language models' performance on diverse tasks and context lengths.
needle-in-a-haystack long-context language models NIAH test synthetic benchmark multi-hop tracing aggregation sequence length task complexity GPT-4 Command-R Yi-34B Mixtral context size