title: zip2zip: Inference-Time Adaptive Vocabularies for Language Models via
    Token Compression
summary: Tokenization efficiency plays a critical role in the performance and cost of
  large language models (LLMs), yet most models rely on static tokenizers
  optimized for general-purpose corpora. These tokenizers' fixed vocabularies
  often fail to adapt to domain- or language-specific inputs, leading to longer
  token sequences and higher computational costs. We introduce zip2zip, a
  framework that enables LLMs to dynamically adjust token vocabulary at inference
  time, allowing for fewer generated tokens and thus faster inference. zip2zip
  consists of three key components: (1) a tokenizer based on Lempel-Ziv-Welch
  (LZW) compression that incrementally compresses tokens into reusable
  "hypertokens" on the fly; (2) an embedding layer that computes embeddings for
  newly formed hypertokens at runtime; and (3) a causal language modeling variant
  that trains the model to operate on hypertokenized, compressed sequences. We
  show that an existing LLM can be zip2zip-fied in 10 GPU-hours via
  parameter-efficient finetuning. The resulting zip2zip LLMs effectively learn to
  use hypertokens at inference time, reducing input and output sequence length by
  20-60\%, with significant improvements in inference latency.
paper_authors._id: 683ea4388be2e40086ea9056
paper_authors.name: Saibo Geng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-03T12:33:50.914Z
paper_authors._id: 5fce0cfeb3dbf216ad31836a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1625748458774-5fce0cfeb3dbf216ad31836a.png
paper_authors.fullname: Saibo-creator
paper_authors.type: user
paper_authors.user: Saibo-creator
paper_authors._id: 683ea4388be2e40086ea9057
paper_authors.name: Nathan Ranchin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-03T09:39:27.313Z
paper_authors._id: 6420afc71ccd411979dc12dc
paper_authors.avatarUrl: /avatars/ee4a89ebc7a0716e21deaebc86e062e6.svg
paper_authors.fullname: nathan ranchin
paper_authors.type: user
paper_authors.user: nathanrchn
paper_authors._id: 683ea4388be2e40086ea9058
paper_authors.name: Yunzhen yao
paper_authors._id: 683ea4388be2e40086ea9059
paper_authors.name: Maxime Peyrard
paper_authors._id: 683ea4388be2e40086ea905a
paper_authors.name: Chris Wendler
paper_authors._id: 683ea4388be2e40086ea905b
paper_authors.name: Michael Gastpar
paper_authors._id: 683ea4388be2e40086ea905c
paper_authors.name: Robert West
paper_title: zip2zip: Inference-Time Adaptive Vocabularies for Language Models via
    Token Compression
paper_summary: Tokenization efficiency plays a critical role in the performance and cost of
  large language models (LLMs), yet most models rely on static tokenizers
  optimized for general-purpose corpora. These tokenizers' fixed vocabularies
  often fail to adapt to domain- or language-specific inputs, leading to longer
  token sequences and higher computational costs. We introduce zip2zip, a
  framework that enables LLMs to dynamically adjust token vocabulary at inference
  time, allowing for fewer generated tokens and thus faster inference. zip2zip
  consists of three key components: (1) a tokenizer based on Lempel-Ziv-Welch
  (LZW) compression that incrementally compresses tokens into reusable
  "hypertokens" on the fly; (2) an embedding layer that computes embeddings for
  newly formed hypertokens at runtime; and (3) a causal language modeling variant
  that trains the model to operate on hypertokenized, compressed sequences. We
  show that an existing LLM can be zip2zip-fied in 10 GPU-hours via
  parameter-efficient finetuning. The resulting zip2zip LLMs effectively learn to
  use hypertokens at inference time, reducing input and output sequence length by
  20-60\%, with significant improvements in inference latency.
paper_ai_summary: A framework called zip2zip dynamically adjusts token vocabulary in LLMs at inference time using LZW compression, reducing token sequence length and improving inference speed.
paper_ai_keywords: LZW compression hypertokens embedding layer causal language modeling parameter-efficient finetuning