title: Trans-Tokenization and Cross-lingual Vocabulary Transfers: Language
    Adaptation of LLMs for Low-Resource NLP
summary: The development of monolingual language models for low and mid-resource
  languages continues to be hindered by the difficulty in sourcing high-quality
  training data. In this study, we present a novel cross-lingual vocabulary
  transfer strategy, trans-tokenization, designed to tackle this challenge and
  enable more efficient language adaptation. Our approach focuses on adapting a
  high-resource monolingual LLM to an unseen target language by initializing the
  token embeddings of the target language using a weighted average of
  semantically similar token embeddings from the source language. For this, we
  leverage a translation resource covering both the source and target languages.
  We validate our method with the Tweeties, a series of trans-tokenized LLMs, and
  demonstrate their competitive performance on various downstream tasks across a
  small but diverse set of languages. Additionally, we introduce Hydra LLMs,
  models with multiple swappable language modeling heads and embedding tables,
  which further extend the capabilities of our trans-tokenization strategy. By
  designing a Hydra LLM based on the multilingual model TowerInstruct, we
  developed a state-of-the-art machine translation model for Tatar, in a
  zero-shot manner, completely bypassing the need for high-quality parallel data.
  This breakthrough is particularly significant for low-resource languages like
  Tatar, where high-quality parallel data is hard to come by. By lowering the
  data and time requirements for training high-quality models, our
  trans-tokenization strategy allows for the development of LLMs for a wider
  range of languages, especially those with limited resources. We hope that our
  work will inspire further research and collaboration in the field of
  cross-lingual vocabulary transfer and contribute to the empowerment of
  languages on a global scale.
paper_authors._id: 66b5ca4332980a2e1ed1e0de
paper_authors.name: François Remy
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-08-09T07:50:28.872Z
paper_authors._id: 5f04e8865d08220171a0ad3f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1647276617786-5f04e8865d08220171a0ad3f.png
paper_authors.fullname: François Remy
paper_authors.type: user
paper_authors.user: FremyCompany
paper_authors._id: 66b5ca4332980a2e1ed1e0df
paper_authors.name: Pieter Delobelle
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-08-09T07:57:57.357Z
paper_authors._id: 5e24810bb2138b1c228eafdc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1617351133158-5e24810bb2138b1c228eafdc.jpeg
paper_authors.fullname: Pieter Delobelle
paper_authors.type: user
paper_authors.user: pdelobelle
paper_authors._id: 66b5ca4332980a2e1ed1e0e0
paper_authors.name: Hayastan Avetisyan
paper_authors._id: 66b5ca4332980a2e1ed1e0e1
paper_authors.name: Alfiya Khabibullina
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-09T10:09:58.523Z
paper_authors._id: 64207825290342c5df8509cd
paper_authors.avatarUrl: /avatars/ae2237bea823677afff0bc4cf30dd924.svg
paper_authors.fullname: Alfiya Khabibullina
paper_authors.type: user
paper_authors.user: justalphie
paper_authors._id: 66b5ca4332980a2e1ed1e0e2
paper_authors.name: Miryam de Lhoneux
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-08-09T07:50:28.872Z
paper_authors._id: 652e8e680b13cc07396e7175
paper_authors.avatarUrl: /avatars/de66c92cca64776e21681451856cf9d3.svg
paper_authors.fullname: Miryam de Lhoneux
paper_authors.type: user
paper_authors.user: mdelhoneux
paper_authors._id: 66b5ca4332980a2e1ed1e0e3
paper_authors.name: Thomas Demeester
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-08-09T07:50:28.872Z
paper_authors._id: 619eaa7857785be19f61e0b2
paper_authors.avatarUrl: /avatars/0ee750e2f79e55d7159fdfb672fecf4f.svg
paper_authors.fullname: Thomas Demeester
paper_authors.type: user
paper_authors.user: tdmeeste
paper_title: Trans-Tokenization and Cross-lingual Vocabulary Transfers: Language
    Adaptation of LLMs for Low-Resource NLP
paper_summary: The development of monolingual language models for low and mid-resource
  languages continues to be hindered by the difficulty in sourcing high-quality
  training data. In this study, we present a novel cross-lingual vocabulary
  transfer strategy, trans-tokenization, designed to tackle this challenge and
  enable more efficient language adaptation. Our approach focuses on adapting a
  high-resource monolingual LLM to an unseen target language by initializing the
  token embeddings of the target language using a weighted average of
  semantically similar token embeddings from the source language. For this, we
  leverage a translation resource covering both the source and target languages.
  We validate our method with the Tweeties, a series of trans-tokenized LLMs, and
  demonstrate their competitive performance on various downstream tasks across a
  small but diverse set of languages. Additionally, we introduce Hydra LLMs,
  models with multiple swappable language modeling heads and embedding tables,
  which further extend the capabilities of our trans-tokenization strategy. By
  designing a Hydra LLM based on the multilingual model TowerInstruct, we
  developed a state-of-the-art machine translation model for Tatar, in a
  zero-shot manner, completely bypassing the need for high-quality parallel data.
  This breakthrough is particularly significant for low-resource languages like
  Tatar, where high-quality parallel data is hard to come by. By lowering the
  data and time requirements for training high-quality models, our
  trans-tokenization strategy allows for the development of LLMs for a wider
  range of languages, especially those with limited resources. We hope that our
  work will inspire further research and collaboration in the field of
  cross-lingual vocabulary transfer and contribute to the empowerment of
  languages on a global scale.
paper_ai_summary: A novel trans-tokenization strategy is introduced to transfer vocabulary from high-resource languages to low-resource languages, enabling competitive performance in various tasks and facilitating zero-shot machine translation for languages like Tatar.
paper_ai_keywords: monolingual language models low-resource languages mid-resource languages cross-lingual vocabulary transfer trans-tokenization token embeddings translation resource Hydra LLMs language modeling heads embedding tables zero-shot machine translation TowerInstruct