How Can We Effectively Expand the Vocabulary of LLMs with 001GB of Target Language Text?
<|reference_start|>How Can We Effectively Expand the Vocabulary of LLMs with 001GB of Target Language Text?: Large language models (LLMs) have shown remarkable capabilities in many languages beyond English. Yet, LLMs require more inference steps when generating non-English text due to their reliance on English-centric tokenizers and vocabulary, resulting in higher usage costs to non-English speakers. Vocabulary expansion with target language tokens is a widely used cross-lingual vocabulary adaptation approach to remedy this issue. Despite its effectiveness in inference speedup, previous work on vocabulary expansion has focused on high-resource settings assuming access to a substantial amount of target language data to effectively initialize the embeddings of the new tokens and adapt the LLM to the target language. However, vocabulary expansion in low-resource settings has yet to be explored. In this paper, we investigate vocabulary expansion in low-resource settings by considering embedding initialization methods and continual pre-training strategies. Through extensive experiments across typologically diverse languages, tasks and models, we establish a set of strategies to perform vocabulary expansion for faster inference, maintaining competitive downstream performance to baselines with only 30K sentences ($\sim$0.01GB text data) from the target language.<|reference_end|>
@article{yamaguchi2024how,
    title={How Can We Effectively Expand the Vocabulary of LLMs with 0.01GB of
  Target Language Text?},
    author={Atsuki Yamaguchi, Aline Villavicencio, Nikolaos Aletras},
    journal={arXiv preprint arXiv:2406.11477},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.11477},
    primaryClass={cs.CL cs.AI}
}