Pretraining Data and Tokenizer for Indic LLM
<|reference_start|>Pretraining Data and Tokenizer for Indic LLM: We present a novel approach to data preparation for developing multilingual Indic large language model. Our meticulous data acquisition spans open-source and proprietary sources, including Common Crawl, Indic books, news articles, and Wikipedia, ensuring a diverse and rich linguistic representation. For each Indic language, we design a custom preprocessing pipeline to effectively eliminate redundant and low-quality text content. Additionally, we perform deduplication on Common Crawl data to address the redundancy present in 70% of the crawled web pages. This study focuses on developing high-quality data, optimizing tokenization for our multilingual dataset for Indic large language models with 3B and 7B parameters, engineered for superior performance in Indic languages. We introduce a novel multilingual tokenizer training strategy, demonstrating our custom-trained Indic tokenizer outperforms the state-of-the-art OpenAI Tiktoken tokenizer, achieving a superior token-to-word ratio for Indic languages.<|reference_end|>
@article{kumar2024pretraining,
    title={Pretraining Data and Tokenizer for Indic LLM},
    author={Rahul Kumar, Shubham Kakde, Divyansh Rajput, Daud Ibrahim, Rishabh
  Nahata, Pidathala Sowjanya, Deepak Kumar},
    journal={arXiv preprint arXiv:2407.12481},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.12481},
    primaryClass={cs.CL}
}