title: DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining
summary: The mixture proportions of pretraining data domains (e.g., Wikipedia, books,
  web text) greatly affect language model (LM) performance. In this paper, we
  propose Domain Reweighting with Minimax Optimization (DoReMi), which first
  trains a small proxy model using group distributionally robust optimization
  (Group DRO) over domains to produce domain weights (mixture proportions)
  without knowledge of downstream tasks. We then resample a dataset with these
  domain weights and train a larger, full-sized model. In our experiments, we use
  DoReMi on a 280M-parameter proxy model to find domain weights for training an
  8B-parameter model (30x larger) more efficiently. On The Pile, DoReMi improves
  perplexity across all domains, even when it downweights a domain. DoReMi
  improves average few-shot downstream accuracy by 6.5% over a baseline model
  trained using The Pile's default domain weights and reaches the baseline
  accuracy with 2.6x fewer training steps. On the GLaM dataset, DoReMi, which has
  no knowledge of downstream tasks, even matches the performance of using domain
  weights tuned on downstream tasks.
paper_authors._id: 646588b3e22ed5ceaa6e3974
paper_authors.name: Sang Michael Xie
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-19T19:04:45.749Z
paper_authors._id: 63d759462e397d9f8e300b9a
paper_authors.avatarUrl: /avatars/8c7514ad5d7fd7c80cbd2c9855162fb1.svg
paper_authors.fullname: Sang Michael Xie
paper_authors.type: user
paper_authors.user: sangmichaelxie
paper_authors._id: 646588b3e22ed5ceaa6e3975
paper_authors.name: Hieu Pham
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-19T19:04:42.598Z
paper_authors._id: 64344bfe7b824748010b6487
paper_authors.avatarUrl: /avatars/e39fceba214e65c825d7e955eb6d03c9.svg
paper_authors.fullname: Hieu Pham
paper_authors.type: user
paper_authors.user: hyhieu
paper_authors._id: 646588b3e22ed5ceaa6e3976
paper_authors.name: Xuanyi Dong
paper_authors._id: 646588b3e22ed5ceaa6e3977
paper_authors.name: Nan Du
paper_authors._id: 646588b3e22ed5ceaa6e3978
paper_authors.name: Hanxiao Liu
paper_authors._id: 646588b3e22ed5ceaa6e3979
paper_authors.name: Yifeng Lu
paper_authors._id: 646588b3e22ed5ceaa6e397a
paper_authors.name: Percy Liang
paper_authors._id: 646588b3e22ed5ceaa6e397b
paper_authors.name: Quoc V. Le
paper_authors._id: 646588b3e22ed5ceaa6e397c
paper_authors.name: Tengyu Ma
paper_authors._id: 646588b3e22ed5ceaa6e397d
paper_authors.name: Adams Wei Yu
paper_title: DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining
paper_summary: The mixture proportions of pretraining data domains (e.g., Wikipedia, books,
  web text) greatly affect language model (LM) performance. In this paper, we
  propose Domain Reweighting with Minimax Optimization (DoReMi), which first
  trains a small proxy model using group distributionally robust optimization
  (Group DRO) over domains to produce domain weights (mixture proportions)
  without knowledge of downstream tasks. We then resample a dataset with these
  domain weights and train a larger, full-sized model. In our experiments, we use
  DoReMi on a 280M-parameter proxy model to find domain weights for training an
  8B-parameter model (30x larger) more efficiently. On The Pile, DoReMi improves
  perplexity across all domains, even when it downweights a domain. DoReMi
  improves average few-shot downstream accuracy by 6.5% over a baseline model
  trained using The Pile's default domain weights and reaches the baseline
  accuracy with 2.6x fewer training steps. On the GLaM dataset, DoReMi, which has
  no knowledge of downstream tasks, even matches the performance of using domain
  weights tuned on downstream tasks.
paper_ai_summary: Domain Reweighting with Minimax Optimization (DoReMi) uses a small proxy model to find optimal domain weights, improving language model performance and efficiency.
paper_ai_keywords: Group Distributionally Robust Optimization Domain Reweighting with Minimax Optimization DoReMi group distributionally robust optimization domain weights few-shot downstream accuracy GLaM dataset