title: Towards Fast Multilingual LLM Inference: Speculative Decoding and
    Specialized Drafters
summary: Large language models (LLMs) have revolutionized natural language processing
  and broadened their applicability across diverse commercial applications.
  However, the deployment of these models is constrained by high inference time
  in multilingual settings. To mitigate this challenge, this paper explores a
  training recipe of an assistant model in speculative decoding, which are
  leveraged to draft and-then its future tokens are verified by the target LLM.
  We show that language-specific draft models, optimized through a targeted
  pretrain-and-finetune strategy, substantially brings a speedup of inference
  time compared to the previous methods. We validate these models across various
  languages in inference time, out-of-domain speedup, and GPT-4o evaluation.
paper_authors._id: 667a2585e737163dcb22f1df
paper_authors.name: Euiin Yi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-25T08:15:32.506Z
paper_authors._id: 66027cead56cd39a44992631
paper_authors.avatarUrl: /avatars/3c8b679bc72f30c547782579e9592bb1.svg
paper_authors.fullname: Euiin Yi
paper_authors.type: user
paper_authors.user: euiin
paper_authors._id: 667a2585e737163dcb22f1e0
paper_authors.name: Taehyeon Kim
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-25T07:33:36.931Z
paper_authors._id: 639f32952e13e54dcbb9bf8d
paper_authors.avatarUrl: /avatars/a797b72363e06cd2222165493bf7e39c.svg
paper_authors.fullname: Taehyeon Kim
paper_authors.type: user
paper_authors.user: Kthyeon
paper_authors._id: 667a2585e737163dcb22f1e1
paper_authors.name: Hongseok Jeung
paper_authors._id: 667a2585e737163dcb22f1e2
paper_authors.name: Du-Seong Chang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-25T08:15:43.224Z
paper_authors._id: 653b44214d105d9696cf5e7a
paper_authors.avatarUrl: /avatars/907509576817ec2199a07d9b42af9c42.svg
paper_authors.fullname: Du-Seong Chang
paper_authors.type: user
paper_authors.user: Du-Seong
paper_authors._id: 667a2585e737163dcb22f1e3
paper_authors.name: Se-Young Yun
paper_title: Towards Fast Multilingual LLM Inference: Speculative Decoding and
    Specialized Drafters
paper_summary: Large language models (LLMs) have revolutionized natural language processing
  and broadened their applicability across diverse commercial applications.
  However, the deployment of these models is constrained by high inference time
  in multilingual settings. To mitigate this challenge, this paper explores a
  training recipe of an assistant model in speculative decoding, which are
  leveraged to draft and-then its future tokens are verified by the target LLM.
  We show that language-specific draft models, optimized through a targeted
  pretrain-and-finetune strategy, substantially brings a speedup of inference
  time compared to the previous methods. We validate these models across various
  languages in inference time, out-of-domain speedup, and GPT-4o evaluation.
paper_ai_summary: A speculative decoding approach with language-specific draft models speeds up inference time for large language models in multilingual settings.
paper_ai_keywords: large language models speculative decoding draft models pretrain-and-finetune strategy GPT-4o evaluation