Cleaner Pretraining Corpus Curation with Neural Web Scraping
<|reference_start|>Cleaner Pretraining Corpus Curation with Neural Web Scraping: The web contains large-scale, diverse, and abundant information to satisfy the information-seeking needs of humans. Through meticulous data collection, preprocessing, and curation, webpages can be used as a fundamental data resource for language model pretraining. However, when confronted with the progressively revolutionized and intricate nature of webpages, rule-based/feature-based web scrapers are becoming increasingly inadequate. This paper presents a simple, fast, and effective Neural web Scraper (NeuScraper) to help extract primary and clean text contents from webpages. Experimental results show that NeuScraper surpasses the baseline scrapers by achieving more than a 20% improvement, demonstrating its potential in extracting higher-quality data to facilitate the language model pretraining. All of the code is available at https://github.com/OpenMatch/NeuScraper.<|reference_end|>
@article{xu2024cleaner,
    title={Cleaner Pretraining Corpus Curation with Neural Web Scraping},
    author={Zhipeng Xu, Zhenghao Liu, Yukun Yan, Zhiyuan Liu, Ge Yu, Chenyan Xiong},
    journal={arXiv preprint arXiv:2402.14652},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.14652},
    primaryClass={cs.CL}
}