title: Facilitating large language model Russian adaptation with Learned
    Embedding Propagation
summary: Rapid advancements of large language model (LLM) technologies led to the
  introduction of powerful open-source instruction-tuned LLMs that have the same
  text generation quality as the state-of-the-art counterparts such as GPT-4.
  While the emergence of such models accelerates the adoption of LLM technologies
  in sensitive-information environments the authors of such models don not
  disclose the training data necessary for replication of the results thus making
  the achievements model-exclusive. Since those open-source models are also
  multilingual this in turn reduces the benefits of training a language specific
  LLMs as improved inference computation efficiency becomes the only guaranteed
  advantage of such costly procedure. More cost-efficient options such as
  vocabulary extension and subsequent continued pre-training are also inhibited
  by the lack of access to high-quality instruction-tuning data since it is the
  major factor behind the resulting LLM task-solving capabilities. To address the
  limitations and cut the costs of the language adaptation pipeline we propose
  Learned Embedding Propagation (LEP). Unlike existing approaches our method has
  lower training data size requirements due to minimal impact on existing LLM
  knowledge which we reinforce using novel ad-hoc embedding propagation procedure
  that allows to skip the instruction-tuning step and instead implant the new
  language knowledge directly into any existing instruct-tuned variant. We
  evaluated four Russian vocabulary adaptations for LLaMa-3-8B and Mistral-7B,
  showing that LEP is competitive with traditional instruction-tuning methods,
  achieving performance comparable to OpenChat 3.5 and LLaMa-3-8B-Instruct, with
  further improvements via self-calibration and continued tuning enhancing
  task-solving capabilities.
paper_authors._id: 6773c24cb272a4f186ec613c
paper_authors.name: Mikhail Tikhomirov
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-12-31T10:07:09.658Z
paper_authors._id: 652cedbdf120598322ae358a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/652cedbdf120598322ae358a/RrxrP0gtQus4SfNwfyAg_.jpeg
paper_authors.fullname: Mikhail
paper_authors.type: user
paper_authors.user: RefalMachine
paper_authors._id: 6773c24cb272a4f186ec613d
paper_authors.name: Daniil Chernyshev
paper_title: Facilitating large language model Russian adaptation with Learned
    Embedding Propagation
paper_summary: Rapid advancements of large language model (LLM) technologies led to the
  introduction of powerful open-source instruction-tuned LLMs that have the same
  text generation quality as the state-of-the-art counterparts such as GPT-4.
  While the emergence of such models accelerates the adoption of LLM technologies
  in sensitive-information environments the authors of such models don not
  disclose the training data necessary for replication of the results thus making
  the achievements model-exclusive. Since those open-source models are also
  multilingual this in turn reduces the benefits of training a language specific
  LLMs as improved inference computation efficiency becomes the only guaranteed
  advantage of such costly procedure. More cost-efficient options such as
  vocabulary extension and subsequent continued pre-training are also inhibited
  by the lack of access to high-quality instruction-tuning data since it is the
  major factor behind the resulting LLM task-solving capabilities. To address the
  limitations and cut the costs of the language adaptation pipeline we propose
  Learned Embedding Propagation (LEP). Unlike existing approaches our method has
  lower training data size requirements due to minimal impact on existing LLM
  knowledge which we reinforce using novel ad-hoc embedding propagation procedure
  that allows to skip the instruction-tuning step and instead implant the new
  language knowledge directly into any existing instruct-tuned variant. We
  evaluated four Russian vocabulary adaptations for LLaMa-3-8B and Mistral-7B,
  showing that LEP is competitive with traditional instruction-tuning methods,
  achieving performance comparable to OpenChat 3.5 and LLaMa-3-8B-Instruct, with
  further improvements via self-calibration and continued tuning enhancing
  task-solving capabilities.
paper_ai_keywords: large language model (LLM) instruction-tuned LLMs text generation quality GPT-4 sensitive-information environments multilingual inference computation efficiency vocabulary extension continuous pre-training high-quality instruction-tuning data task-solving capabilities Learned Embedding Propagation (LEP) embedding propagation procedure minimal impact on existing LLM knowledge instruction-tuning step LLaMa-3-8B Mistral-7B OpenChat 3.5 LLaMa-3-8B-Instruct self-calibration continued tuning