title: Adapting LLMs to Hebrew: Unveiling DictaLM 2.0 with Enhanced Vocabulary
    and Instruction Capabilities
summary: Training large language models (LLMs) in low-resource languages such as
  Hebrew poses unique challenges. In this paper, we introduce DictaLM2.0 and
  DictaLM2.0-Instruct, two LLMs derived from the Mistral model, trained on a
  substantial corpus of approximately 200 billion tokens in both Hebrew and
  English. Adapting a pre-trained model to a new language involves specialized
  techniques that differ significantly from training a model from scratch or
  further training existing models on well-resourced languages such as English.
  We outline these novel training methodologies, which facilitate effective
  learning and adaptation to the linguistic properties of Hebrew. Additionally,
  we fine-tuned DictaLM2.0-Instruct on a comprehensive instruct dataset to
  enhance its performance on task-specific instructions. To rigorously evaluate
  our models, we introduce a new benchmark suite for Hebrew LLM evaluation,
  covering a diverse set of tasks including Question Answering, Sentiment
  Analysis, Winograd Schema Challenge, Translation, and Summarization. Our work
  not only addresses the intricacies of training LLMs in low-resource languages
  but also proposes a framework that can be leveraged for adapting other LLMs to
  various non-English languages, contributing to the broader field of
  multilingual NLP.
paper_authors._id: 668e5182f59574a8ec209239
paper_authors.name: Shaltiel Shmidman
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-10T10:20:09.683Z
paper_authors._id: 641c500c21964f8f6d456bb5
paper_authors.avatarUrl: /avatars/668dfce780e1681234f9116822592119.svg
paper_authors.fullname: Shaltiel Shmidman
paper_authors.type: user
paper_authors.user: Shaltiel
paper_authors._id: 668e5182f59574a8ec20923a
paper_authors.name: Avi Shmidman
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-10T10:51:36.897Z
paper_authors._id: 650246cb61e3c2daa042a5f5
paper_authors.avatarUrl: /avatars/f6422dc2934c009c4ecb3ee266f9f135.svg
paper_authors.fullname: Avi Shmidman
paper_authors.type: user
paper_authors.user: avishmidman
paper_authors._id: 668e5182f59574a8ec20923b
paper_authors.name: Amir DN Cohen
paper_authors._id: 668e5182f59574a8ec20923c
paper_authors.name: Moshe Koppel
paper_title: Adapting LLMs to Hebrew: Unveiling DictaLM 2.0 with Enhanced Vocabulary
    and Instruction Capabilities
paper_summary: Training large language models (LLMs) in low-resource languages such as
  Hebrew poses unique challenges. In this paper, we introduce DictaLM2.0 and
  DictaLM2.0-Instruct, two LLMs derived from the Mistral model, trained on a
  substantial corpus of approximately 200 billion tokens in both Hebrew and
  English. Adapting a pre-trained model to a new language involves specialized
  techniques that differ significantly from training a model from scratch or
  further training existing models on well-resourced languages such as English.
  We outline these novel training methodologies, which facilitate effective
  learning and adaptation to the linguistic properties of Hebrew. Additionally,
  we fine-tuned DictaLM2.0-Instruct on a comprehensive instruct dataset to
  enhance its performance on task-specific instructions. To rigorously evaluate
  our models, we introduce a new benchmark suite for Hebrew LLM evaluation,
  covering a diverse set of tasks including Question Answering, Sentiment
  Analysis, Winograd Schema Challenge, Translation, and Summarization. Our work
  not only addresses the intricacies of training LLMs in low-resource languages
  but also proposes a framework that can be leveraged for adapting other LLMs to
  various non-English languages, contributing to the broader field of
  multilingual NLP.
paper_ai_summary: DictaLM2.0 and DictaLM2.0-Instruct, derived from the Mistral model, are fine-tuned on Hebrew and English using novel methodologies to address challenges in low-resource language training, evaluated on a comprehensive Hebrew benchmark suite.
paper_ai_keywords: LLMs DictaLM2.0 DictaLM2.0-Instruct Mistral model pre-trained model fine-tuning instruct dataset Hebrew LLM evaluation benchmark suite Question Answering Sentiment Analysis Winograd Schema Challenge Translation Summarization multilingual NLP