title: Rephrasing the Web: A Recipe for Compute and Data-Efficient Language
    Modeling
summary: Large language models are trained on massive scrapes of the web, which are
  often unstructured, noisy, and poorly phrased. Current scaling laws show that
  learning from such data requires an abundance of both compute and data, which
  grows with the size of the model being trained. This is infeasible both because
  of the large compute costs and duration associated with pre-training, and the
  impending scarcity of high-quality data on the web. In this work, we propose
  Web Rephrase Augmented Pre-training (WRAP) that uses an
  off-the-shelf instruction-tuned model prompted to paraphrase documents on the
  web in specific styles such as "like Wikipedia" or in "question-answer format"
  to jointly pre-train LLMs on real and synthetic rephrases. First, we show that
  using WRAP on the C4 dataset, which is naturally noisy, speeds up pre-training
  by sim3x. At the same pre-training compute budget, it improves perplexity by
  more than 10% on average across different subsets of the Pile, and improves
  zero-shot question answer accuracy across 13 tasks by more than 2%. Second, we
  investigate the impact of the re-phrasing style on the performance of the
  model, offering insights into how the composition of the training data can
  impact the performance of LLMs in OOD settings. Our gains are attributed to the
  fact that re-phrased synthetic data has higher utility than just real data
  because it (i) incorporates style diversity that closely reflects downstream
  evaluation style, and (ii) has higher 'quality' than web-scraped data.
paper_authors._id: 65b871a0fc836d9cd5a59ee1
paper_authors.name: Pratyush Maini
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-30T08:06:16.882Z
paper_authors._id: 632c826d1d303f5f9acf5917
paper_authors.avatarUrl: /avatars/defca406f2c4f027a7b1be9a5ebbbd01.svg
paper_authors.fullname: Pratyush Maini
paper_authors.type: user
paper_authors.user: pratyushmaini
paper_authors._id: 65b871a0fc836d9cd5a59ee2
paper_authors.name: Skyler Seto
paper_authors._id: 65b871a0fc836d9cd5a59ee3
paper_authors.name: He Bai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-30T09:00:04.116Z
paper_authors._id: 5f024bd8ff69163f6f59e775
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1652111948107-5f024bd8ff69163f6f59e775.png
paper_authors.fullname: Richard He Bai
paper_authors.type: user
paper_authors.user: richardbaihe
paper_authors._id: 65b871a0fc836d9cd5a59ee4
paper_authors.name: David Grangier
paper_authors._id: 65b871a0fc836d9cd5a59ee5
paper_authors.name: Yizhe Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-30T08:13:36.937Z
paper_authors._id: 5e64858c87403103f9f1055d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5dd96eb166059660ed1ee413/96-oNO2OLVkAFOojMaV0T.jpeg
paper_authors.fullname: Yizhe Zhang
paper_authors.type: user
paper_authors.user: yizhezhang
paper_authors._id: 65b871a0fc836d9cd5a59ee6
paper_authors.name: Navdeep Jaitly
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-30T08:13:43.714Z
paper_authors._id: 631119df88942700629f4aab
paper_authors.avatarUrl: /avatars/251c20a8fd48831a8c0cc0d198a42455.svg
paper_authors.fullname: Navdeep Jaitly
paper_authors.type: user
paper_authors.user: ndjaitly
paper_title: Rephrasing the Web: A Recipe for Compute and Data-Efficient Language
    Modeling
paper_summary: Large language models are trained on massive scrapes of the web, which are
  often unstructured, noisy, and poorly phrased. Current scaling laws show that
  learning from such data requires an abundance of both compute and data, which
  grows with the size of the model being trained. This is infeasible both because
  of the large compute costs and duration associated with pre-training, and the
  impending scarcity of high-quality data on the web. In this work, we propose
  Web Rephrase Augmented Pre-training (WRAP) that uses an
  off-the-shelf instruction-tuned model prompted to paraphrase documents on the
  web in specific styles such as "like Wikipedia" or in "question-answer format"
  to jointly pre-train LLMs on real and synthetic rephrases. First, we show that
  using WRAP on the C4 dataset, which is naturally noisy, speeds up pre-training
  by sim3x. At the same pre-training compute budget, it improves perplexity by
  more than 10% on average across different subsets of the Pile, and improves
  zero-shot question answer accuracy across 13 tasks by more than 2%. Second, we
  investigate the impact of the re-phrasing style on the performance of the
  model, offering insights into how the composition of the training data can
  impact the performance of LLMs in OOD settings. Our gains are attributed to the
  fact that re-phrased synthetic data has higher utility than just real data
  because it (i) incorporates style diversity that closely reflects downstream
  evaluation style, and (ii) has higher 'quality' than web-scraped data.
paper_ai_summary: Web Rephrase Augmented Pre-training (WRAP) improves large language model pre-training efficiency and performance by incorporating synthetic rephrases styled like Wikipedia or question-answer format into the training dataset.
paper_ai_keywords: instruction-tuned model rephrasing styles Wikipedia style question-answer format C4 dataset perplexity zero-shot question answer accuracy out-of-distribution (OOD) settings synthetic data style diversity data quality