title: RULER: What's the Real Context Size of Your Long-Context Language
    Models?
summary: The needle-in-a-haystack (NIAH) test, which examines the ability to retrieve
  a piece of information (the "needle") from long distractor texts (the
  "haystack"), has been widely adopted to evaluate long-context language models
  (LMs). However, this simple retrieval-based test is indicative of only a
  superficial form of long-context understanding. To provide a more comprehensive
  evaluation of long-context LMs, we create a new synthetic benchmark RULER with
  flexible configurations for customized sequence length and task complexity.
  RULER expands upon the vanilla NIAH test to encompass variations with diverse
  types and quantities of needles. Moreover, RULER introduces new task categories
  multi-hop tracing and aggregation to test behaviors beyond searching from
  context. We evaluate ten long-context LMs with 13 representative tasks in
  RULER. Despite achieving nearly perfect accuracy in the vanilla NIAH test, all
  models exhibit large performance drops as the context length increases. While
  these models all claim context sizes of 32K tokens or greater, only four models
  (GPT-4, Command-R, Yi-34B, and Mixtral) can maintain satisfactory performance
  at the length of 32K. Our analysis of Yi-34B, which supports context length of
  200K, reveals large room for improvement as we increase input length and task
  complexity. We open source RULER to spur comprehensive evaluation of
  long-context LMs.
paper_authors._id: 66173e869c0e16d020e31ef7
paper_authors.name: Cheng-Ping Hsieh
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-11T20:47:44.740Z
paper_authors._id: 651c80b534a090d301fa211a
paper_authors.avatarUrl: /avatars/ec34ca524616894b62695d025d4fd6d1.svg
paper_authors.fullname: Cheng-Ping Hsieh
paper_authors.type: user
paper_authors.user: hsiehjackson
paper_authors._id: 66173e869c0e16d020e31ef8
paper_authors.name: Simeng Sun
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-12T05:01:59.073Z
paper_authors._id: 62cd08c1ff01ad1c9e509bc9
paper_authors.avatarUrl: /avatars/2dc8a0c381962e8c35c6571c406b46ab.svg
paper_authors.fullname: Simeng Sun
paper_authors.type: user
paper_authors.user: simsun131
paper_authors._id: 66173e869c0e16d020e31ef9
paper_authors.name: Samuel Kriman
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T12:40:54.419Z
paper_authors._id: 63b884a1aaa8cf17f0975921
paper_authors.avatarUrl: /avatars/fc3c807b68d190ea5606ca2c30c3fad6.svg
paper_authors.fullname: Samuel Kriman
paper_authors.type: user
paper_authors.user: nv-sam
paper_authors._id: 66173e869c0e16d020e31efa
paper_authors.name: Shantanu Acharya
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-11T14:54:49.981Z
paper_authors._id: 626c5978f451470f86251117
paper_authors.avatarUrl: /avatars/c2ecdf2d4eb8ca1c14ba427933cc38b0.svg
paper_authors.fullname: Shantanu Acharya
paper_authors.type: user
paper_authors.user: shantanuacharya
paper_authors._id: 66173e869c0e16d020e31efb
paper_authors.name: Dima Rekesh
paper_authors._id: 66173e869c0e16d020e31efc
paper_authors.name: Fei Jia
paper_authors._id: 66173e869c0e16d020e31efd
paper_authors.name: Boris Ginsburg
paper_title: RULER: What's the Real Context Size of Your Long-Context Language
    Models?
paper_summary: The needle-in-a-haystack (NIAH) test, which examines the ability to retrieve
  a piece of information (the "needle") from long distractor texts (the
  "haystack"), has been widely adopted to evaluate long-context language models
  (LMs). However, this simple retrieval-based test is indicative of only a
  superficial form of long-context understanding. To provide a more comprehensive
  evaluation of long-context LMs, we create a new synthetic benchmark RULER with
  flexible configurations for customized sequence length and task complexity.
  RULER expands upon the vanilla NIAH test to encompass variations with diverse
  types and quantities of needles. Moreover, RULER introduces new task categories
  multi-hop tracing and aggregation to test behaviors beyond searching from
  context. We evaluate ten long-context LMs with 13 representative tasks in
  RULER. Despite achieving nearly perfect accuracy in the vanilla NIAH test, all
  models exhibit large performance drops as the context length increases. While
  these models all claim context sizes of 32K tokens or greater, only four models
  (GPT-4, Command-R, Yi-34B, and Mixtral) can maintain satisfactory performance
  at the length of 32K. Our analysis of Yi-34B, which supports context length of
  200K, reveals large room for improvement as we increase input length and task
  complexity. We open source RULER to spur comprehensive evaluation of
  long-context LMs.
paper_ai_keywords: long-context language models NIAH test retrieval-based test synthetic benchmark RULER multi-hop tracing aggregation vanilla NIAH test sequence length task complexity performance drops context length context size context sizes token count input length open source RULER comprehensive evaluation