Rephrasing the Web: A Recipe for Compute and Data-Efficient Language
  Modeling
Large language models are trained on massive scrapes of the web, which are
often unstructured, noisy, and poorly phrased. Current scaling laws show that
learning from such data requires an abundance of both compute and data, which
grows with the size of the model being trained. This is infeasible both because
of the large compute costs and duration associated with pre-training, and the
impending scarcity of high-quality data on the web. In this work, we propose
Web Rephrase Augmented Pre-training (WRAP) that uses an
off-the-shelf instruction-tuned model prompted to paraphrase documents on the
web in specific styles such as "like Wikipedia" or in "question-answer format"
to jointly pre-train LLMs on real and synthetic rephrases. First, we show that
using WRAP on the C4 dataset, which is naturally noisy, speeds up pre-training
by sim3x. At the same pre-training compute budget, it improves perplexity by
more than 10% on average across different subsets of the Pile, and improves
zero-shot question answer accuracy across 13 tasks by more than 2%. Second, we
investigate the impact of the re-phrasing style on the performance of the
model, offering insights into how the composition of the training data can
impact the performance of LLMs in OOD settings. Our gains are attributed to the
fact that re-phrased synthetic data has higher utility than just real data
because it (i) incorporates style diversity that closely reflects downstream
evaluation style, and (ii) has higher 'quality' than web-scraped data.
65b871a0fc836d9cd5a59ee1
Pratyush Maini
claimed_verified
2024-01-30T08:06:16.882Z
632c826d1d303f5f9acf5917
/avatars/defca406f2c4f027a7b1be9a5ebbbd01.svg
Pratyush Maini
user
pratyushmaini
65b871a0fc836d9cd5a59ee2
Skyler Seto
65b871a0fc836d9cd5a59ee3
He Bai
claimed_verified
2024-01-30T09:00:04.116Z
5f024bd8ff69163f6f59e775
https://cdn-avatars.huggingface.co/v1/production/uploads/1652111948107-5f024bd8ff69163f6f59e775.png
Richard He Bai
user
richardbaihe
65b871a0fc836d9cd5a59ee4
David Grangier
65b871a0fc836d9cd5a59ee5
Yizhe Zhang
admin_assigned
2024-01-30T08:13:36.937Z
5e64858c87403103f9f1055d
https://cdn-avatars.huggingface.co/v1/production/uploads/5dd96eb166059660ed1ee413/96-oNO2OLVkAFOojMaV0T.jpeg
Yizhe Zhang
user
yizhezhang
65b871a0fc836d9cd5a59ee6
Navdeep Jaitly
admin_assigned
2024-01-30T08:13:43.714Z
631119df88942700629f4aab
/avatars/251c20a8fd48831a8c0cc0d198a42455.svg
Navdeep Jaitly
user
ndjaitly
Rephrasing the Web: A Recipe for Compute and Data-Efficient Language
  Modeling
Large language models are trained on massive scrapes of the web, which are
often unstructured, noisy, and poorly phrased. Current scaling laws show that
learning from such data requires an abundance of both compute and data, which
grows with the size of the model being trained. This is infeasible both because
of the large compute costs and duration associated with pre-training, and the
impending scarcity of high-quality data on the web. In this work, we propose
Web Rephrase Augmented Pre-training (WRAP) that uses an
off-the-shelf instruction-tuned model prompted to paraphrase documents on the
web in specific styles such as "like Wikipedia" or in "question-answer format"
to jointly pre-train LLMs on real and synthetic rephrases. First, we show that
using WRAP on the C4 dataset, which is naturally noisy, speeds up pre-training
by sim3x. At the same pre-training compute budget, it improves perplexity by
more than 10% on average across different subsets of the Pile, and improves
zero-shot question answer accuracy across 13 tasks by more than 2%. Second, we
investigate the impact of the re-phrasing style on the performance of the
model, offering insights into how the composition of the training data can
impact the performance of LLMs in OOD settings. Our gains are attributed to the
fact that re-phrased synthetic data has higher utility than just real data
because it (i) incorporates style diversity that closely reflects downstream
evaluation style, and (ii) has higher 'quality' than web-scraped data.
Web Rephrase Augmented Pre-training (WRAP) improves large language model pre-training efficiency and performance by incorporating synthetic rephrases styled like Wikipedia or question-answer format into the training dataset.
instruction-tuned model rephrasing styles Wikipedia style question-answer format C4 dataset perplexity zero-shot question answer accuracy out-of-distribution (OOD) settings synthetic data style diversity data quality