title: Evaluating Tokenizer Performance of Large Language Models Across
    Official Indian Languages
summary: Large Language Models (LLMs) based on transformer architectures have
  revolutionized a variety of domains, with tokenization playing a pivotal role
  in their pre-processing and fine-tuning stages. In multilingual models,
  particularly those tailored for Indic languages, effective tokenization is
  crucial for optimizing performance. This paper presents a comprehensive
  evaluation of tokenizers used by 12 LLMs across all 22 official languages of
  India, with a focus on comparing the efficiency of their tokenization
  processes. We employed the Normalized Sequence Length (NSL) as a key metric in
  our analysis. Our findings reveal that the SUTRA tokenizer outperforms all
  other models, including several Indic-specific models, excelling in 14
  languages. Notable insights include the SUTRA tokenizer's superior handling of
  Indic languages, GPT-4o's advancement over its predecessor GPT-4 in processing
  Indian languages, and the limited performance of Project Indus in certain
  languages. This study underscores the critical importance of developing
  targeted tokenization strategies for multilingual and Indic-centric models,
  laying the groundwork for future improvements in tokenizer design to enhance
  linguistic coverage and model efficiency.
paper_authors._id: 673d599340b6ff437a9ae9d8
paper_authors.name: S. Tamang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-20T10:08:33.948Z
paper_authors._id: 65d3518c40e07f7dbf7ca772
paper_authors.avatarUrl: /avatars/018e780ebeacc8440487d5f550574906.svg
paper_authors.fullname: SAGAR TAMANG
paper_authors.type: user
paper_authors.user: tamang0000
paper_authors._id: 673d599340b6ff437a9ae9d9
paper_authors.name: D. J. Bora
paper_title: Evaluating Tokenizer Performance of Large Language Models Across
    Official Indian Languages
paper_summary: Large Language Models (LLMs) based on transformer architectures have
  revolutionized a variety of domains, with tokenization playing a pivotal role
  in their pre-processing and fine-tuning stages. In multilingual models,
  particularly those tailored for Indic languages, effective tokenization is
  crucial for optimizing performance. This paper presents a comprehensive
  evaluation of tokenizers used by 12 LLMs across all 22 official languages of
  India, with a focus on comparing the efficiency of their tokenization
  processes. We employed the Normalized Sequence Length (NSL) as a key metric in
  our analysis. Our findings reveal that the SUTRA tokenizer outperforms all
  other models, including several Indic-specific models, excelling in 14
  languages. Notable insights include the SUTRA tokenizer's superior handling of
  Indic languages, GPT-4o's advancement over its predecessor GPT-4 in processing
  Indian languages, and the limited performance of Project Indus in certain
  languages. This study underscores the critical importance of developing
  targeted tokenization strategies for multilingual and Indic-centric models,
  laying the groundwork for future improvements in tokenizer design to enhance
  linguistic coverage and model efficiency.
paper_ai_summary: A comprehensive evaluation finds SUTRA tokenizer excels for tokenizing Indic languages across various LLMs, surpassing competitors in handling multiple Indian languages.
paper_ai_keywords: Large Language Models transformer architectures tokenization multilingual models Indic languages Normalized Sequence Length SUTRA tokenizer GPT-4o Project Indus targeted tokenization strategies