Optimizing LLMs for Italian: Reducing Token Fertility and Enhancing
  Efficiency Through Vocabulary Adaptation
The number of pretrained Large Language Models (LLMs) is increasing steadily,
though the majority are designed predominantly for the English language. While
state-of-the-art LLMs can handle other languages, due to language contamination
or some degree of multilingual pretraining data, they are not optimized for
non-English languages, leading to inefficient encoding (high token "fertility")
and slower inference speed. In this work, we thoroughly compare a variety of
vocabulary adaptation techniques for optimizing English LLMs for the Italian
language, and put forward Semantic Alignment Vocabulary Adaptation (SAVA), a
novel method that leverages neural mapping for vocabulary substitution. SAVA
achieves competitive performance across multiple downstream tasks, enhancing
grounded alignment strategies. We adapt two LLMs: Mistral-7b-v0.1, reducing
token fertility by 25\%, and Llama-3.1-8B, optimizing the vocabulary and
reducing the number of parameters by 1 billion. We show that, following the
adaptation of the vocabulary, these models can recover their performance with a
relatively limited stage of continual training on the target language. Finally,
we test the capabilities of the adapted models on various multi-choice and
generative tasks.
680cb47facf3c58ba7e13b73
Luca Moroni
claimed_verified
2025-04-30T07:56:48.529Z
652cf31268f1d7d1d2a11f19
https://cdn-avatars.huggingface.co/v1/production/uploads/652cf31268f1d7d1d2a11f19/rwtuOCAAunfKTBg2spLlc.jpeg
luca m
user
iperbole
680cb47facf3c58ba7e13b74
Giovanni Puccetti
admin_assigned
2025-04-29T12:43:49.935Z
60c72bad2c6fd4f486d25b4b
https://cdn-avatars.huggingface.co/v1/production/uploads/60c72bad2c6fd4f486d25b4b/H-hZfI-td4Ngyn7rORyx2.jpeg
Giovanni Puccetti
user
gpucce
680cb47facf3c58ba7e13b75
Pere-Lluis Huguet Cabot
admin_assigned
2025-04-29T12:43:56.014Z
5f04e7545d08220171a0ad3e
https://cdn-avatars.huggingface.co/v1/production/uploads/5f04e7545d08220171a0ad3e/M0dWlakxnLakQlZl5iOq6.jpeg
Pere-Lluis Huguet Cabot
user
PereLluis13
680cb47facf3c58ba7e13b76
Andrei Stefan Bejgu
admin_assigned
2025-04-29T12:44:02.000Z
633ee9704be90e06da1eb774
https://cdn-avatars.huggingface.co/v1/production/uploads/633ee9704be90e06da1eb774/xU6gXNHkBub0hAQb9RRsR.jpeg
Andrei Stefan Bejgu
user
andreim14
680cb47facf3c58ba7e13b77
Edoardo Barba
admin_assigned
2025-04-29T12:44:08.675Z
614dc83d8d64cfc347c28e42
https://cdn-avatars.huggingface.co/v1/production/uploads/614dc83d8d64cfc347c28e42/FM3R1AHeO1I7P5gA3GD_V.jpeg
Edoardo Barba
user
edobobo
680cb47facf3c58ba7e13b78
Alessio Miaschi
claimed_verified
2025-04-28T12:50:18.302Z
62e3fe792a8df5b22feed15e
https://cdn-avatars.huggingface.co/v1/production/uploads/62e3fe792a8df5b22feed15e/iy8J4oGhW8Nr9QAgoH8ES.jpeg
Alessio Miaschi
user
alemiaschi
680cb47facf3c58ba7e13b79
Felice Dell'Orletta
admin_assigned
2025-04-29T12:44:14.334Z
663ca4637527788a57f4d73f
/avatars/797e68ee21492a753df82a9f7c2e800e.svg
Felice Dell'Orletta
user
Falice10
680cb47facf3c58ba7e13b7a
Andrea Esuli
admin_assigned
2025-04-29T12:44:20.194Z
631859d6e562b32a0d6ef727
/avatars/199805fc5207eece784bb02e9406c8ae.svg
Andrea Esuli
user
aesuli
680cb47facf3c58ba7e13b7b
Roberto Navigli
admin_assigned
2025-04-29T12:44:26.827Z
63845c7c99292a80134bc784
https://cdn-avatars.huggingface.co/v1/production/uploads/63845c7c99292a80134bc784/eKOPJMjm8F-JUKT_mi7Dv.png
Roberto Navigli
user
navigli
Optimizing LLMs for Italian: Reducing Token Fertility and Enhancing
  Efficiency Through Vocabulary Adaptation
The number of pretrained Large Language Models (LLMs) is increasing steadily,
though the majority are designed predominantly for the English language. While
state-of-the-art LLMs can handle other languages, due to language contamination
or some degree of multilingual pretraining data, they are not optimized for
non-English languages, leading to inefficient encoding (high token "fertility")
and slower inference speed. In this work, we thoroughly compare a variety of
vocabulary adaptation techniques for optimizing English LLMs for the Italian
language, and put forward Semantic Alignment Vocabulary Adaptation (SAVA), a
novel method that leverages neural mapping for vocabulary substitution. SAVA
achieves competitive performance across multiple downstream tasks, enhancing
grounded alignment strategies. We adapt two LLMs: Mistral-7b-v0.1, reducing
token fertility by 25\%, and Llama-3.1-8B, optimizing the vocabulary and
reducing the number of parameters by 1 billion. We show that, following the
adaptation of the vocabulary, these models can recover their performance with a
relatively limited stage of continual training on the target language. Finally,
we test the capabilities of the adapted models on various multi-choice and
generative tasks.
SAVA, a novel vocabulary adaptation technique using neural mapping, optimizes English LLMs for Italian, reducing token fertility and improving performance across various tasks with minimal additional training.
Large Language Models LLMs Semantic Alignment Vocabulary Adaptation SAVA vocabulary adaptation neural mapping token fertility Mistral-7b-v0.1 Llama-3.1-8B continual training multi-choice tasks generative tasks