DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining
The mixture proportions of pretraining data domains (e.g., Wikipedia, books,
web text) greatly affect language model (LM) performance. In this paper, we
propose Domain Reweighting with Minimax Optimization (DoReMi), which first
trains a small proxy model using group distributionally robust optimization
(Group DRO) over domains to produce domain weights (mixture proportions)
without knowledge of downstream tasks. We then resample a dataset with these
domain weights and train a larger, full-sized model. In our experiments, we use
DoReMi on a 280M-parameter proxy model to find domain weights for training an
8B-parameter model (30x larger) more efficiently. On The Pile, DoReMi improves
perplexity across all domains, even when it downweights a domain. DoReMi
improves average few-shot downstream accuracy by 6.5% over a baseline model
trained using The Pile's default domain weights and reaches the baseline
accuracy with 2.6x fewer training steps. On the GLaM dataset, DoReMi, which has
no knowledge of downstream tasks, even matches the performance of using domain
weights tuned on downstream tasks.
646588b3e22ed5ceaa6e3974
Sang Michael Xie
claimed_verified
2023-05-19T19:04:45.749Z
63d759462e397d9f8e300b9a
/avatars/8c7514ad5d7fd7c80cbd2c9855162fb1.svg
Sang Michael Xie
user
sangmichaelxie
646588b3e22ed5ceaa6e3975
Hieu Pham
claimed_verified
2023-05-19T19:04:42.598Z
64344bfe7b824748010b6487
/avatars/e39fceba214e65c825d7e955eb6d03c9.svg
Hieu Pham
user
hyhieu
646588b3e22ed5ceaa6e3976
Xuanyi Dong
646588b3e22ed5ceaa6e3977
Nan Du
646588b3e22ed5ceaa6e3978
Hanxiao Liu
646588b3e22ed5ceaa6e3979
Yifeng Lu
646588b3e22ed5ceaa6e397a
Percy Liang
646588b3e22ed5ceaa6e397b
Quoc V. Le
646588b3e22ed5ceaa6e397c
Tengyu Ma
646588b3e22ed5ceaa6e397d
Adams Wei Yu
DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining
The mixture proportions of pretraining data domains (e.g., Wikipedia, books,
web text) greatly affect language model (LM) performance. In this paper, we
propose Domain Reweighting with Minimax Optimization (DoReMi), which first
trains a small proxy model using group distributionally robust optimization
(Group DRO) over domains to produce domain weights (mixture proportions)
without knowledge of downstream tasks. We then resample a dataset with these
domain weights and train a larger, full-sized model. In our experiments, we use
DoReMi on a 280M-parameter proxy model to find domain weights for training an
8B-parameter model (30x larger) more efficiently. On The Pile, DoReMi improves
perplexity across all domains, even when it downweights a domain. DoReMi
improves average few-shot downstream accuracy by 6.5% over a baseline model
trained using The Pile's default domain weights and reaches the baseline
accuracy with 2.6x fewer training steps. On the GLaM dataset, DoReMi, which has
no knowledge of downstream tasks, even matches the performance of using domain
weights tuned on downstream tasks.
Domain Reweighting with Minimax Optimization (DoReMi) uses a small proxy model to find optimal domain weights, improving language model performance and efficiency.
Group Distributionally Robust Optimization Domain Reweighting with Minimax Optimization DoReMi group distributionally robust optimization domain weights few-shot downstream accuracy GLaM dataset