Order Matters in the Presence of Dataset Imbalance for Multilingual
  Learning
In this paper, we empirically study the optimization dynamics of multi-task
learning, particularly focusing on those that govern a collection of tasks with
significant data imbalance. We present a simple yet effective method of
pre-training on high-resource tasks, followed by fine-tuning on a mixture of
high/low-resource tasks. We provide a thorough empirical study and analysis of
this method's benefits showing that it achieves consistent improvements
relative to the performance trade-off profile of standard static weighting. We
analyze under what data regimes this method is applicable and show its
improvements empirically in neural machine translation (NMT) and multi-lingual
language modeling.
6577efc7dd2996f01ae7077b
Dami Choi
admin_assigned
2023-12-12T08:58:53.917Z
641cacf592cd2530299819aa
/avatars/7e0ee9d19f05deb320f359b9625a0526.svg
Dami Choi
user
choidami
6577efc7dd2996f01ae7077c
Derrick Xin
admin_assigned
2023-12-12T08:59:00.772Z
634cea1605f736dff37abe84
/avatars/6c199de4b6e7fd1320f47eeecd1310f2.svg
Derrick Xin
user
Derr
6577efc7dd2996f01ae7077d
Hamid Dadkhahi
6577efc7dd2996f01ae7077e
Justin Gilmer
6577efc7dd2996f01ae7077f
Ankush Garg
admin_assigned
2023-12-12T08:59:12.353Z
609bfde2138207cc2f2f3a8c
/avatars/8bb0198645bf8aa4e473bab3b1d76b20.svg
garg
user
ankushgarg
6577efc7dd2996f01ae70780
Orhan Firat
6577efc7dd2996f01ae70781
Chih-Kuan Yeh
6577efc7dd2996f01ae70782
Andrew M. Dai
admin_assigned
2023-12-12T08:59:29.785Z
64c04c0b5e03515457ed72a3
/avatars/dcbfca63b32a8d5c575232316c01dbe7.svg
Andrew Dai
user
PenguinPush
6577efc7dd2996f01ae70783
Behrooz Ghorbani
Order Matters in the Presence of Dataset Imbalance for Multilingual
  Learning
In this paper, we empirically study the optimization dynamics of multi-task
learning, particularly focusing on those that govern a collection of tasks with
significant data imbalance. We present a simple yet effective method of
pre-training on high-resource tasks, followed by fine-tuning on a mixture of
high/low-resource tasks. We provide a thorough empirical study and analysis of
this method's benefits showing that it achieves consistent improvements
relative to the performance trade-off profile of standard static weighting. We
analyze under what data regimes this method is applicable and show its
improvements empirically in neural machine translation (NMT) and multi-lingual
language modeling.
A method of pre-training on high-resource tasks followed by fine-tuning on a mix of high/low-resource tasks improves performance in multi-task learning with data imbalance, particularly in neural machine translation and multi-lingual language modeling.
multi-task learning data imbalance pre-training fine-tuning static weighting neural machine translation multi-lingual language modeling