title: AdaptiVocab: Enhancing LLM Efficiency in Focused Domains through
    Lightweight Vocabulary Adaptation
summary: Large Language Models (LLMs) have shown impressive versatility as general
  purpose models. However, their broad applicability comes at a high-cost
  computational overhead, particularly in auto-regressive decoding where each
  step requires a forward pass. In domain-specific settings, general-purpose
  capabilities are unnecessary and can be exchanged for efficiency. In this work,
  we take a novel perspective on domain adaptation, reducing latency and
  computational costs by adapting the vocabulary to focused domains of interest.
  We introduce AdaptiVocab, an end-to-end approach for vocabulary adaptation,
  designed to enhance LLM efficiency in low-resource domains. AdaptiVocab can be
  applied to any tokenizer and architecture, modifying the vocabulary by
  replacing tokens with domain-specific n-gram-based tokens, thereby reducing the
  number of tokens required for both input processing and output generation.
  AdaptiVocab initializes new n-token embeddings using an exponentially weighted
  combination of existing embeddings and employs a lightweight fine-tuning phase
  that can be efficiently performed on a single GPU. We evaluate two 7B LLMs
  across three niche domains, assessing efficiency, generation quality, and
  end-task performance. Our results show that AdaptiVocab reduces token usage by
  over 25% without compromising performance
paper_authors._id: 67ea363dd13d75fc156ec498
paper_authors.name: Itay Nakash
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-31T08:24:09.634Z
paper_authors._id: 671f8106d677d3a764a6f9a5
paper_authors.avatarUrl: /avatars/90b4b00058aac30c060c5eac8debb1c7.svg
paper_authors.fullname: itay nakash
paper_authors.type: user
paper_authors.user: itaynakash
paper_authors._id: 67ea363dd13d75fc156ec499
paper_authors.name: Nitay Calderon
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-31T08:24:15.760Z
paper_authors._id: 62d6a0c18faee0ac953c51fa
paper_authors.avatarUrl: /avatars/ca818cebdb089a8d853c5bc4d5e0987b.svg
paper_authors.fullname: Nitay Calderon
paper_authors.type: user
paper_authors.user: nitay
paper_authors._id: 67ea363dd13d75fc156ec49a
paper_authors.name: Eyal Ben David
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-31T08:24:21.914Z
paper_authors._id: 6645fc650e6706053171ce51
paper_authors.avatarUrl: /avatars/54b03ac6939d4b8943606b12b979ce52.svg
paper_authors.fullname: Eyal Ben-David
paper_authors.type: user
paper_authors.user: eyalbd
paper_authors._id: 67ea363dd13d75fc156ec49b
paper_authors.name: Elad Hoffer
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-31T08:24:27.895Z
paper_authors._id: 630480fa6dbbb80f16352ee3
paper_authors.avatarUrl: /avatars/f39ce2fe96a578f42a57e3bfe3a2d137.svg
paper_authors.fullname: Elad Hoffer
paper_authors.type: user
paper_authors.user: ehoffer
paper_authors._id: 67ea363dd13d75fc156ec49c
paper_authors.name: Roi Reichart
paper_title: AdaptiVocab: Enhancing LLM Efficiency in Focused Domains through
    Lightweight Vocabulary Adaptation
paper_summary: Large Language Models (LLMs) have shown impressive versatility as general
  purpose models. However, their broad applicability comes at a high-cost
  computational overhead, particularly in auto-regressive decoding where each
  step requires a forward pass. In domain-specific settings, general-purpose
  capabilities are unnecessary and can be exchanged for efficiency. In this work,
  we take a novel perspective on domain adaptation, reducing latency and
  computational costs by adapting the vocabulary to focused domains of interest.
  We introduce AdaptiVocab, an end-to-end approach for vocabulary adaptation,
  designed to enhance LLM efficiency in low-resource domains. AdaptiVocab can be
  applied to any tokenizer and architecture, modifying the vocabulary by
  replacing tokens with domain-specific n-gram-based tokens, thereby reducing the
  number of tokens required for both input processing and output generation.
  AdaptiVocab initializes new n-token embeddings using an exponentially weighted
  combination of existing embeddings and employs a lightweight fine-tuning phase
  that can be efficiently performed on a single GPU. We evaluate two 7B LLMs
  across three niche domains, assessing efficiency, generation quality, and
  end-task performance. Our results show that AdaptiVocab reduces token usage by
  over 25% without compromising performance
paper_ai_summary: AdaptiVocab, a novel vocabulary adaptation method, enhances the efficiency of Large Language Models in domain-specific applications without sacrificing performance by reducing token usage.
paper_ai_keywords: Large Language Models LLMs domain adaptation auto-regressive decoding vocabulary adaptation AdaptiVocab n-gram-based tokens token embeddings fine-tuning end-to-end approach