title: Optimizing LLMs for Italian: Reducing Token Fertility and Enhancing
    Efficiency Through Vocabulary Adaptation
summary: The number of pretrained Large Language Models (LLMs) is increasing steadily,
  though the majority are designed predominantly for the English language. While
  state-of-the-art LLMs can handle other languages, due to language contamination
  or some degree of multilingual pretraining data, they are not optimized for
  non-English languages, leading to inefficient encoding (high token "fertility")
  and slower inference speed. In this work, we thoroughly compare a variety of
  vocabulary adaptation techniques for optimizing English LLMs for the Italian
  language, and put forward Semantic Alignment Vocabulary Adaptation (SAVA), a
  novel method that leverages neural mapping for vocabulary substitution. SAVA
  achieves competitive performance across multiple downstream tasks, enhancing
  grounded alignment strategies. We adapt two LLMs: Mistral-7b-v0.1, reducing
  token fertility by 25\%, and Llama-3.1-8B, optimizing the vocabulary and
  reducing the number of parameters by 1 billion. We show that, following the
  adaptation of the vocabulary, these models can recover their performance with a
  relatively limited stage of continual training on the target language. Finally,
  we test the capabilities of the adapted models on various multi-choice and
  generative tasks.
paper_authors._id: 680cb47facf3c58ba7e13b73
paper_authors.name: Luca Moroni
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-30T07:56:48.529Z
paper_authors._id: 652cf31268f1d7d1d2a11f19
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/652cf31268f1d7d1d2a11f19/rwtuOCAAunfKTBg2spLlc.jpeg
paper_authors.fullname: luca m
paper_authors.type: user
paper_authors.user: iperbole
paper_authors._id: 680cb47facf3c58ba7e13b74
paper_authors.name: Giovanni Puccetti
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-29T12:43:49.935Z
paper_authors._id: 60c72bad2c6fd4f486d25b4b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60c72bad2c6fd4f486d25b4b/H-hZfI-td4Ngyn7rORyx2.jpeg
paper_authors.fullname: Giovanni Puccetti
paper_authors.type: user
paper_authors.user: gpucce
paper_authors._id: 680cb47facf3c58ba7e13b75
paper_authors.name: Pere-Lluis Huguet Cabot
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-29T12:43:56.014Z
paper_authors._id: 5f04e7545d08220171a0ad3e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5f04e7545d08220171a0ad3e/M0dWlakxnLakQlZl5iOq6.jpeg
paper_authors.fullname: Pere-Lluis Huguet Cabot
paper_authors.type: user
paper_authors.user: PereLluis13
paper_authors._id: 680cb47facf3c58ba7e13b76
paper_authors.name: Andrei Stefan Bejgu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-29T12:44:02.000Z
paper_authors._id: 633ee9704be90e06da1eb774
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/633ee9704be90e06da1eb774/xU6gXNHkBub0hAQb9RRsR.jpeg
paper_authors.fullname: Andrei Stefan Bejgu
paper_authors.type: user
paper_authors.user: andreim14
paper_authors._id: 680cb47facf3c58ba7e13b77
paper_authors.name: Edoardo Barba
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-29T12:44:08.675Z
paper_authors._id: 614dc83d8d64cfc347c28e42
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/614dc83d8d64cfc347c28e42/FM3R1AHeO1I7P5gA3GD_V.jpeg
paper_authors.fullname: Edoardo Barba
paper_authors.type: user
paper_authors.user: edobobo
paper_authors._id: 680cb47facf3c58ba7e13b78
paper_authors.name: Alessio Miaschi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-28T12:50:18.302Z
paper_authors._id: 62e3fe792a8df5b22feed15e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62e3fe792a8df5b22feed15e/iy8J4oGhW8Nr9QAgoH8ES.jpeg
paper_authors.fullname: Alessio Miaschi
paper_authors.type: user
paper_authors.user: alemiaschi
paper_authors._id: 680cb47facf3c58ba7e13b79
paper_authors.name: Felice Dell'Orletta
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-29T12:44:14.334Z
paper_authors._id: 663ca4637527788a57f4d73f
paper_authors.avatarUrl: /avatars/797e68ee21492a753df82a9f7c2e800e.svg
paper_authors.fullname: Felice Dell'Orletta
paper_authors.type: user
paper_authors.user: Falice10
paper_authors._id: 680cb47facf3c58ba7e13b7a
paper_authors.name: Andrea Esuli
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-29T12:44:20.194Z
paper_authors._id: 631859d6e562b32a0d6ef727
paper_authors.avatarUrl: /avatars/199805fc5207eece784bb02e9406c8ae.svg
paper_authors.fullname: Andrea Esuli
paper_authors.type: user
paper_authors.user: aesuli
paper_authors._id: 680cb47facf3c58ba7e13b7b
paper_authors.name: Roberto Navigli
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-29T12:44:26.827Z
paper_authors._id: 63845c7c99292a80134bc784
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63845c7c99292a80134bc784/eKOPJMjm8F-JUKT_mi7Dv.png
paper_authors.fullname: Roberto Navigli
paper_authors.type: user
paper_authors.user: navigli
paper_title: Optimizing LLMs for Italian: Reducing Token Fertility and Enhancing
    Efficiency Through Vocabulary Adaptation
paper_summary: The number of pretrained Large Language Models (LLMs) is increasing steadily,
  though the majority are designed predominantly for the English language. While
  state-of-the-art LLMs can handle other languages, due to language contamination
  or some degree of multilingual pretraining data, they are not optimized for
  non-English languages, leading to inefficient encoding (high token "fertility")
  and slower inference speed. In this work, we thoroughly compare a variety of
  vocabulary adaptation techniques for optimizing English LLMs for the Italian
  language, and put forward Semantic Alignment Vocabulary Adaptation (SAVA), a
  novel method that leverages neural mapping for vocabulary substitution. SAVA
  achieves competitive performance across multiple downstream tasks, enhancing
  grounded alignment strategies. We adapt two LLMs: Mistral-7b-v0.1, reducing
  token fertility by 25\%, and Llama-3.1-8B, optimizing the vocabulary and
  reducing the number of parameters by 1 billion. We show that, following the
  adaptation of the vocabulary, these models can recover their performance with a
  relatively limited stage of continual training on the target language. Finally,
  we test the capabilities of the adapted models on various multi-choice and
  generative tasks.
paper_ai_summary: SAVA, a novel vocabulary adaptation technique using neural mapping, optimizes English LLMs for Italian, reducing token fertility and improving performance across various tasks with minimal additional training.
paper_ai_keywords: Large Language Models LLMs Semantic Alignment Vocabulary Adaptation SAVA vocabulary adaptation neural mapping token fertility Mistral-7b-v0.1 Llama-3.1-8B continual training multi-choice tasks generative tasks