title: Data Mixture Inference: What do BPE Tokenizers Reveal about their
    Training Data?
summary: The pretraining data of today's strongest language models is opaque. In
  particular, little is known about the proportions of various domains or
  languages represented. In this work, we tackle a task which we call data
  mixture inference, which aims to uncover the distributional make-up of training
  data. We introduce a novel attack based on a previously overlooked source of
  information -- byte-pair encoding (BPE) tokenizers, used by the vast majority
  of modern language models. Our key insight is that the ordered list of merge
  rules learned by a BPE tokenizer naturally reveals information about the token
  frequencies in its training data: the first merge is the most common byte pair,
  the second is the most common pair after merging the first token, and so on.
  Given a tokenizer's merge list along with data samples for each category of
  interest, we formulate a linear program that solves for the proportion of each
  category in the tokenizer's training set. Importantly, to the extent to which
  tokenizer training data is representative of the pretraining data, we
  indirectly learn about the pretraining data. In controlled experiments, we show
  that our attack recovers mixture ratios with high precision for tokenizers
  trained on known mixtures of natural languages, programming languages, and data
  sources. We then apply our approach to off-the-shelf tokenizers released with
  recent LMs. We confirm much publicly disclosed information about these models,
  and also make several new inferences: GPT-4o's tokenizer is much more
  multilingual than its predecessors, training on 39% non-English data; Llama3
  extends GPT-3.5's tokenizer primarily for multilingual (48%) use; GPT-3.5's and
  Claude's tokenizers are trained on predominantly code (~60%). We hope our work
  sheds light on current design practices for pretraining data, and inspires
  continued research into data mixture inference for LMs.
paper_authors._id: 66a0f22917555c302003b936
paper_authors.name: Jonathan Hayase
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-26T12:18:19.787Z
paper_authors._id: 62f68be78465979f627b94b6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62f68be78465979f627b94b6/AfygEy8RaanUbWBkIxbey.jpeg
paper_authors.fullname: Jonathan Hayase
paper_authors.type: user
paper_authors.user: Jhayase
paper_authors._id: 66a0f22917555c302003b937
paper_authors.name: Alisa Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-26T12:18:55.764Z
paper_authors._id: 62449da46b55ef458d430371
paper_authors.avatarUrl: /avatars/b95954a88c7b15524a3d7776a4abe083.svg
paper_authors.fullname: Alisa Liu
paper_authors.type: user
paper_authors.user: alisawuffles
paper_authors._id: 66a0f22917555c302003b938
paper_authors.name: Yejin Choi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-26T12:19:14.024Z
paper_authors._id: 64d42729f63b01b7f676b176
paper_authors.avatarUrl: /avatars/52e54bdd6a1fb6c774a40cd70f3d7925.svg
paper_authors.fullname: Yejin Choi
paper_authors.type: user
paper_authors.user: yejinchoinka
paper_authors._id: 66a0f22917555c302003b939
paper_authors.name: Sewoong Oh
paper_authors._id: 66a0f22917555c302003b93a
paper_authors.name: Noah A. Smith
paper_title: Data Mixture Inference: What do BPE Tokenizers Reveal about their
    Training Data?
paper_summary: The pretraining data of today's strongest language models is opaque. In
  particular, little is known about the proportions of various domains or
  languages represented. In this work, we tackle a task which we call data
  mixture inference, which aims to uncover the distributional make-up of training
  data. We introduce a novel attack based on a previously overlooked source of
  information -- byte-pair encoding (BPE) tokenizers, used by the vast majority
  of modern language models. Our key insight is that the ordered list of merge
  rules learned by a BPE tokenizer naturally reveals information about the token
  frequencies in its training data: the first merge is the most common byte pair,
  the second is the most common pair after merging the first token, and so on.
  Given a tokenizer's merge list along with data samples for each category of
  interest, we formulate a linear program that solves for the proportion of each
  category in the tokenizer's training set. Importantly, to the extent to which
  tokenizer training data is representative of the pretraining data, we
  indirectly learn about the pretraining data. In controlled experiments, we show
  that our attack recovers mixture ratios with high precision for tokenizers
  trained on known mixtures of natural languages, programming languages, and data
  sources. We then apply our approach to off-the-shelf tokenizers released with
  recent LMs. We confirm much publicly disclosed information about these models,
  and also make several new inferences: GPT-4o's tokenizer is much more
  multilingual than its predecessors, training on 39% non-English data; Llama3
  extends GPT-3.5's tokenizer primarily for multilingual (48%) use; GPT-3.5's and
  Claude's tokenizers are trained on predominantly code (~60%). We hope our work
  sheds light on current design practices for pretraining data, and inspires
  continued research into data mixture inference for LMs.
paper_ai_summary: Data mixture inference using byte-pair encoding (BPE) tokenizers uncovers the distribution of training data in language models, revealing proportions of various domains and languages.
paper_ai_keywords: byte-pair encoding BPE tokenizers token frequencies merge rules linear program natural languages programming languages data sources multilingual code