Web Rephrase Augmented Pre-training (WRAP) enhances language model training efficiency by transforming documents into structured formats.
text
Web Rephrase Augmented Pre-training (WRAP) enhances language model training efficiency by transforming documents into structured formats.
new_line
new_line
Key aspects:
text
Key aspects:
new_line
* Utilizes an instruction-tuned model to rephrase web content into styles such as Wikipedia or Q/A, creating a blend of synthetic and real data for training.
text
* Utilizes an instruction-tuned model to rephrase web content into styles such as Wikipedia or Q/A, creating a blend of synthetic and real data for training.
new_line
* Demonstrated improvements of over 10% better perplexity, alongside more than 2% increase in zero-shot question-answering accuracy.
text
* Demonstrated improvements of over 10% better perplexity, alongside more than 2% increase in zero-shot question-answering accuracy.
new_line
new_line
Congrats to the authors for their work!
text
Congrats to the authors for their work!
new_line
new_line
Paper: 
text
Paper: 
Rephrasing the Web: A Recipe for Compute and Data-Efficient Language
  Modeling (2401.16380)
https://huggingface.co/papers/2401.16380
2401.16380
paper
resource
https://huggingface.co/papers/2401.16380
Web Rephrase Augmented Pre-training (WRAP) enhances language model training efficiency by transforming documents into structured formats.

Key aspects:
* Utilizes an instruction-tuned model to rephrase web content into styles such as Wikipedia or Q/A, creating a blend of synthetic and real data for training.
* Demonstrated improvements of over 10% better perplexity, alongside more than 2% increase in zero-shot question-answering accuracy.

Congrats to the authors for their work!

Paper: https://huggingface.co/papers/2401.16380
❤️
osanseviero clem lunarflu BroCodeR Alexandro14 kotyKD abdullah
657217faabb25ed8aedd5e48
https://cdn-avatars.huggingface.co/v1/production/uploads/657217faabb25ed8aedd5e48/UUHAXeGtOnQBXFD3nYtf2.jpeg
Vlad Bogolin
vladbogo
user