title: UnifiedCrawl: Aggregated Common Crawl for Affordable Adaptation of LLMs
    on Low-Resource Languages
summary: Large language models (LLMs) under-perform on low-resource languages due to
  limited training data. We present a method to efficiently collect text data for
  low-resource languages from the entire Common Crawl corpus. Our approach,
  UnifiedCrawl, filters and extracts common crawl using minimal compute
  resources, yielding mono-lingual datasets much larger than previously available
  sources. We demonstrate that leveraging this data to fine-tuning multilingual
  LLMs via efficient adapter methods (QLoRA) significantly boosts performance on
  the low-resource language, while minimizing VRAM usage. Our experiments show
  large improvements in language modeling perplexity and an increase in few-shot
  prompting scores. Our work and released source code provide an affordable
  approach to improve LLMs for low-resource languages using consumer hardware.
  Our source code is available here at
  https://github.com/bethelmelesse/unifiedcrawl.
paper_authors._id: 674061fa1b8664981af12c10
paper_authors.name: Bethel Melesse Tessema
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-22T13:19:03.955Z
paper_authors._id: 63668e73cbf2cf329187b45d
paper_authors.avatarUrl: /avatars/814ebbcc9d251c6bc526309714a9baef.svg
paper_authors.fullname: Bethel Melesse Tessema
paper_authors.type: user
paper_authors.user: bethelmelesse
paper_authors._id: 674061fa1b8664981af12c11
paper_authors.name: Akhil Kedia
paper_authors._id: 674061fa1b8664981af12c12
paper_authors.name: Tae-Sun Chung
paper_title: UnifiedCrawl: Aggregated Common Crawl for Affordable Adaptation of LLMs
    on Low-Resource Languages
paper_summary: Large language models (LLMs) under-perform on low-resource languages due to
  limited training data. We present a method to efficiently collect text data for
  low-resource languages from the entire Common Crawl corpus. Our approach,
  UnifiedCrawl, filters and extracts common crawl using minimal compute
  resources, yielding mono-lingual datasets much larger than previously available
  sources. We demonstrate that leveraging this data to fine-tuning multilingual
  LLMs via efficient adapter methods (QLoRA) significantly boosts performance on
  the low-resource language, while minimizing VRAM usage. Our experiments show
  large improvements in language modeling perplexity and an increase in few-shot
  prompting scores. Our work and released source code provide an affordable
  approach to improve LLMs for low-resource languages using consumer hardware.
  Our source code is available here at
  https://github.com/bethelmelesse/unifiedcrawl.
paper_ai_keywords: UnifiedCrawl common crawl mono-lingual datasets fine-tuning multilingual LLMs efficient adapter methods QLoRA language modeling perplexity few-shot prompting scores