content.raw: Web Rephrase Augmented Pre-training (WRAP) enhances language model training efficiency by transforming documents into structured formats.
content.type: text
content.value: Web Rephrase Augmented Pre-training (WRAP) enhances language model training efficiency by transforming documents into structured formats.
content.type: new_line
content.type: new_line
content.raw: Key aspects:
content.type: text
content.value: Key aspects:
content.type: new_line
content.raw: * Utilizes an instruction-tuned model to rephrase web content into styles such as Wikipedia or Q/A, creating a blend of synthetic and real data for training.
content.type: text
content.value: * Utilizes an instruction-tuned model to rephrase web content into styles such as Wikipedia or Q/A, creating a blend of synthetic and real data for training.
content.type: new_line
content.raw: * Demonstrated improvements of over 10% better perplexity, alongside more than 2% increase in zero-shot question-answering accuracy.
content.type: text
content.value: * Demonstrated improvements of over 10% better perplexity, alongside more than 2% increase in zero-shot question-answering accuracy.
content.type: new_line
content.type: new_line
content.raw: Congrats to the authors for their work!
content.type: text
content.value: Congrats to the authors for their work!
content.type: new_line
content.type: new_line
content.raw: Paper: 
content.type: text
content.value: Paper: 
content.label: Rephrasing the Web: A Recipe for Compute and Data-Efficient Language
    Modeling (2401.16380)
content.raw: https://huggingface.co/papers/2401.16380
content.id: 2401.16380
content.type: paper
content.type: resource
content.url: https://huggingface.co/papers/2401.16380
rawContent: Web Rephrase Augmented Pre-training (WRAP) enhances language model training efficiency by transforming documents into structured formats.
  
  Key aspects:
  * Utilizes an instruction-tuned model to rephrase web content into styles such as Wikipedia or Q/A, creating a blend of synthetic and real data for training.
  * Demonstrated improvements of over 10% better perplexity, alongside more than 2% increase in zero-shot question-answering accuracy.
  
  Congrats to the authors for their work!
  
  Paper: https://huggingface.co/papers/2401.16380
reactions.reaction: ❤️
reactions.users: osanseviero clem lunarflu BroCodeR Alexandro14 kotyKD abdullah
commentators._id: 657217faabb25ed8aedd5e48
commentators.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/657217faabb25ed8aedd5e48/UUHAXeGtOnQBXFD3nYtf2.jpeg
commentators.fullname: Vlad Bogolin
commentators.name: vladbogo
commentators.type: user