zip2zip: Inference-Time Adaptive Vocabularies for Language Models via
  Token Compression
Tokenization efficiency plays a critical role in the performance and cost of
large language models (LLMs), yet most models rely on static tokenizers
optimized for general-purpose corpora. These tokenizers' fixed vocabularies
often fail to adapt to domain- or language-specific inputs, leading to longer
token sequences and higher computational costs. We introduce zip2zip, a
framework that enables LLMs to dynamically adjust token vocabulary at inference
time, allowing for fewer generated tokens and thus faster inference. zip2zip
consists of three key components: (1) a tokenizer based on Lempel-Ziv-Welch
(LZW) compression that incrementally compresses tokens into reusable
"hypertokens" on the fly; (2) an embedding layer that computes embeddings for
newly formed hypertokens at runtime; and (3) a causal language modeling variant
that trains the model to operate on hypertokenized, compressed sequences. We
show that an existing LLM can be zip2zip-fied in 10 GPU-hours via
parameter-efficient finetuning. The resulting zip2zip LLMs effectively learn to
use hypertokens at inference time, reducing input and output sequence length by
20-60\%, with significant improvements in inference latency.
683ea4388be2e40086ea9056
Saibo Geng
claimed_verified
2025-06-03T12:33:50.914Z
5fce0cfeb3dbf216ad31836a
https://cdn-avatars.huggingface.co/v1/production/uploads/1625748458774-5fce0cfeb3dbf216ad31836a.png
Saibo-creator
user
Saibo-creator
683ea4388be2e40086ea9057
Nathan Ranchin
claimed_verified
2025-06-03T09:39:27.313Z
6420afc71ccd411979dc12dc
/avatars/ee4a89ebc7a0716e21deaebc86e062e6.svg
nathan ranchin
user
nathanrchn
683ea4388be2e40086ea9058
Yunzhen yao
683ea4388be2e40086ea9059
Maxime Peyrard
683ea4388be2e40086ea905a
Chris Wendler
683ea4388be2e40086ea905b
Michael Gastpar
683ea4388be2e40086ea905c
Robert West
zip2zip: Inference-Time Adaptive Vocabularies for Language Models via
  Token Compression
Tokenization efficiency plays a critical role in the performance and cost of
large language models (LLMs), yet most models rely on static tokenizers
optimized for general-purpose corpora. These tokenizers' fixed vocabularies
often fail to adapt to domain- or language-specific inputs, leading to longer
token sequences and higher computational costs. We introduce zip2zip, a
framework that enables LLMs to dynamically adjust token vocabulary at inference
time, allowing for fewer generated tokens and thus faster inference. zip2zip
consists of three key components: (1) a tokenizer based on Lempel-Ziv-Welch
(LZW) compression that incrementally compresses tokens into reusable
"hypertokens" on the fly; (2) an embedding layer that computes embeddings for
newly formed hypertokens at runtime; and (3) a causal language modeling variant
that trains the model to operate on hypertokenized, compressed sequences. We
show that an existing LLM can be zip2zip-fied in 10 GPU-hours via
parameter-efficient finetuning. The resulting zip2zip LLMs effectively learn to
use hypertokens at inference time, reducing input and output sequence length by
20-60\%, with significant improvements in inference latency.
A framework called zip2zip dynamically adjusts token vocabulary in LLMs at inference time using LZW compression, reducing token sequence length and improving inference speed.
LZW compression hypertokens embedding layer causal language modeling parameter-efficient finetuning