Adapting LLMs to Hebrew: Unveiling DictaLM 2.0 with Enhanced Vocabulary
  and Instruction Capabilities
Training large language models (LLMs) in low-resource languages such as
Hebrew poses unique challenges. In this paper, we introduce DictaLM2.0 and
DictaLM2.0-Instruct, two LLMs derived from the Mistral model, trained on a
substantial corpus of approximately 200 billion tokens in both Hebrew and
English. Adapting a pre-trained model to a new language involves specialized
techniques that differ significantly from training a model from scratch or
further training existing models on well-resourced languages such as English.
We outline these novel training methodologies, which facilitate effective
learning and adaptation to the linguistic properties of Hebrew. Additionally,
we fine-tuned DictaLM2.0-Instruct on a comprehensive instruct dataset to
enhance its performance on task-specific instructions. To rigorously evaluate
our models, we introduce a new benchmark suite for Hebrew LLM evaluation,
covering a diverse set of tasks including Question Answering, Sentiment
Analysis, Winograd Schema Challenge, Translation, and Summarization. Our work
not only addresses the intricacies of training LLMs in low-resource languages
but also proposes a framework that can be leveraged for adapting other LLMs to
various non-English languages, contributing to the broader field of
multilingual NLP.
668e5182f59574a8ec209239
Shaltiel Shmidman
claimed_verified
2024-07-10T10:20:09.683Z
641c500c21964f8f6d456bb5
/avatars/668dfce780e1681234f9116822592119.svg
Shaltiel Shmidman
user
Shaltiel
668e5182f59574a8ec20923a
Avi Shmidman
admin_assigned
2024-07-10T10:51:36.897Z
650246cb61e3c2daa042a5f5
/avatars/f6422dc2934c009c4ecb3ee266f9f135.svg
Avi Shmidman
user
avishmidman
668e5182f59574a8ec20923b
Amir DN Cohen
668e5182f59574a8ec20923c
Moshe Koppel
Adapting LLMs to Hebrew: Unveiling DictaLM 2.0 with Enhanced Vocabulary
  and Instruction Capabilities
Training large language models (LLMs) in low-resource languages such as
Hebrew poses unique challenges. In this paper, we introduce DictaLM2.0 and
DictaLM2.0-Instruct, two LLMs derived from the Mistral model, trained on a
substantial corpus of approximately 200 billion tokens in both Hebrew and
English. Adapting a pre-trained model to a new language involves specialized
techniques that differ significantly from training a model from scratch or
further training existing models on well-resourced languages such as English.
We outline these novel training methodologies, which facilitate effective
learning and adaptation to the linguistic properties of Hebrew. Additionally,
we fine-tuned DictaLM2.0-Instruct on a comprehensive instruct dataset to
enhance its performance on task-specific instructions. To rigorously evaluate
our models, we introduce a new benchmark suite for Hebrew LLM evaluation,
covering a diverse set of tasks including Question Answering, Sentiment
Analysis, Winograd Schema Challenge, Translation, and Summarization. Our work
not only addresses the intricacies of training LLMs in low-resource languages
but also proposes a framework that can be leveraged for adapting other LLMs to
various non-English languages, contributing to the broader field of
multilingual NLP.
DictaLM2.0 and DictaLM2.0-Instruct, derived from the Mistral model, are fine-tuned on Hebrew and English using novel methodologies to address challenges in low-resource language training, evaluated on a comprehensive Hebrew benchmark suite.
LLMs DictaLM2.0 DictaLM2.0-Instruct Mistral model pre-trained model fine-tuning instruct dataset Hebrew LLM evaluation benchmark suite Question Answering Sentiment Analysis Winograd Schema Challenge Translation Summarization multilingual NLP