title: Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion
    Tokens
summary: Are n-gram language models still relevant in this era of neural large
  language models (LLMs)? Our answer is yes, and we show their values in both
  text analysis and improving neural LLMs. Yet this necessitates modernizing
  n-gram models in two aspects. First, we train them at the same data scale as
  neural LLMs -- 1.4 trillion tokens. This is the largest n-gram model ever
  built. Second, existing n-gram models use small n which hinders their
  performance; we instead allow n to be arbitrarily large, by introducing a new
  infty-gram LM with backoff. Instead of pre-computing n-gram count tables
  (which would be very expensive), we develop an engine named infini-gram --
  powered by suffix arrays -- that can compute infty-gram (as well as n-gram
  with arbitrary n) probabilities with millisecond-level latency. The
  infty-gram framework and infini-gram engine enable us to conduct many novel
  and interesting analyses of human-written and machine-generated text: we find
  that the infty-gram LM has fairly high accuracy for next-token prediction
  (47%), and can complement neural LLMs to greatly reduce their language modeling
  perplexities. When analyzing machine-generated text, we also observe
  irregularities in the machine--infty-gram agreement level with respect to
  the suffix length, which indicates deficiencies in neural LLM pretraining and
  the positional embeddings of Transformers. We open-source our infini-gram
  engine in the hopes of enabling more study on how to best use verbatim
  information retrieved from large text corpora.
paper_authors._id: 65baf35d87b68ba54f65c487
paper_authors.name: Jiacheng Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-01T09:30:14.988Z
paper_authors._id: 635f46d1928a42bc95cfcf7c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/635f46d1928a42bc95cfcf7c/5KF8aLiDCJdl7B1SdJ-7V.png
paper_authors.fullname: Jiacheng Liu
paper_authors.type: user
paper_authors.user: liujch1998
paper_authors._id: 65baf35d87b68ba54f65c488
paper_authors.name: Sewon Min
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-01T09:33:54.936Z
paper_authors._id: 63a76d0de27a6dbd485fe863
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63a76d0de27a6dbd485fe863/qJJwHOuvyQGq1o0KscOF_.jpeg
paper_authors.fullname: Sewon Min
paper_authors.type: user
paper_authors.user: sewon
paper_authors._id: 65baf35d87b68ba54f65c489
paper_authors.name: Luke Zettlemoyer
paper_authors._id: 65baf35d87b68ba54f65c48a
paper_authors.name: Yejin Choi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-01T09:34:24.458Z
paper_authors._id: 64d42729f63b01b7f676b176
paper_authors.avatarUrl: /avatars/52e54bdd6a1fb6c774a40cd70f3d7925.svg
paper_authors.fullname: Yejin Choi
paper_authors.type: user
paper_authors.user: yejinchoinka
paper_authors._id: 65baf35d87b68ba54f65c48b
paper_authors.name: Hannaneh Hajishirzi
paper_title: Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion
    Tokens
paper_summary: Are n-gram language models still relevant in this era of neural large
  language models (LLMs)? Our answer is yes, and we show their values in both
  text analysis and improving neural LLMs. Yet this necessitates modernizing
  n-gram models in two aspects. First, we train them at the same data scale as
  neural LLMs -- 1.4 trillion tokens. This is the largest n-gram model ever
  built. Second, existing n-gram models use small n which hinders their
  performance; we instead allow n to be arbitrarily large, by introducing a new
  infty-gram LM with backoff. Instead of pre-computing n-gram count tables
  (which would be very expensive), we develop an engine named infini-gram --
  powered by suffix arrays -- that can compute infty-gram (as well as n-gram
  with arbitrary n) probabilities with millisecond-level latency. The
  infty-gram framework and infini-gram engine enable us to conduct many novel
  and interesting analyses of human-written and machine-generated text: we find
  that the infty-gram LM has fairly high accuracy for next-token prediction
  (47%), and can complement neural LLMs to greatly reduce their language modeling
  perplexities. When analyzing machine-generated text, we also observe
  irregularities in the machine--infty-gram agreement level with respect to
  the suffix length, which indicates deficiencies in neural LLM pretraining and
  the positional embeddings of Transformers. We open-source our infini-gram
  engine in the hopes of enabling more study on how to best use verbatim
  information retrieved from large text corpora.
paper_ai_keywords: n-gram language models neural large language models (LLMs) $\infty$-gram LM backoff infini-izing engine suffix arrays next-token prediction language modeling perplexities machine-generated text Transformer positional embeddings