title: Long-range Language Modeling with Self-retrieval
summary: Retrieval-augmented language models (LMs) have received much attention
  recently. However, typically the retriever is not trained jointly as a native
  component of the LM, but added to an already-pretrained LM, which limits the
  ability of the LM and the retriever to adapt to one another. In this work, we
  propose the Retrieval-Pretrained Transformer (RPT), an architecture and
  training procedure for jointly training a retrieval-augmented LM from scratch
  for the task of modeling long texts. Given a recently generated text chunk in a
  long document, the LM computes query representations, which are then used to
  retrieve earlier chunks in the document, located potentially tens of thousands
  of tokens before. Information from retrieved chunks is fused into the LM
  representations to predict the next target chunk. We train the retriever
  component with a semantic objective, where the goal is to retrieve chunks that
  increase the probability of the next chunk, according to a reference LM. We
  evaluate RPT on four long-range language modeling tasks, spanning books, code,
  and mathematical writing, and demonstrate that RPT improves retrieval quality
  and subsequently perplexity across the board compared to strong baselines.
paper_authors._id: 64991fad42d585137eba2df4
paper_authors.name: Ohad Rubin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-26T08:47:00.919Z
paper_authors._id: 61d0bec503bc10eb8e1c297d
paper_authors.avatarUrl: /avatars/7ae0cbe5abbfc056de47640f1592595e.svg
paper_authors.fullname: Ohad Rubin
paper_authors.type: user
paper_authors.user: iohadrubin
paper_authors._id: 64991fad42d585137eba2df5
paper_authors.name: Jonathan Berant
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-26T08:48:34.915Z
paper_authors._id: 60811c9a4ad99100d63ce36c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1619119757620-60811c9a4ad99100d63ce36c.png
paper_authors.fullname: Jonathan Berant
paper_authors.type: user
paper_authors.user: joberant
paper_title: Long-range Language Modeling with Self-retrieval
paper_summary: Retrieval-augmented language models (LMs) have received much attention
  recently. However, typically the retriever is not trained jointly as a native
  component of the LM, but added to an already-pretrained LM, which limits the
  ability of the LM and the retriever to adapt to one another. In this work, we
  propose the Retrieval-Pretrained Transformer (RPT), an architecture and
  training procedure for jointly training a retrieval-augmented LM from scratch
  for the task of modeling long texts. Given a recently generated text chunk in a
  long document, the LM computes query representations, which are then used to
  retrieve earlier chunks in the document, located potentially tens of thousands
  of tokens before. Information from retrieved chunks is fused into the LM
  representations to predict the next target chunk. We train the retriever
  component with a semantic objective, where the goal is to retrieve chunks that
  increase the probability of the next chunk, according to a reference LM. We
  evaluate RPT on four long-range language modeling tasks, spanning books, code,
  and mathematical writing, and demonstrate that RPT improves retrieval quality
  and subsequently perplexity across the board compared to strong baselines.
paper_ai_keywords: Retrieval-augmented language models (LMs) Retrieval-Pretrained Transformer (RPT) Joint training Query representations Semantic objective Long texts Long-range language modeling tasks Perplexity