title: Order Matters in the Presence of Dataset Imbalance for Multilingual
    Learning
summary: In this paper, we empirically study the optimization dynamics of multi-task
  learning, particularly focusing on those that govern a collection of tasks with
  significant data imbalance. We present a simple yet effective method of
  pre-training on high-resource tasks, followed by fine-tuning on a mixture of
  high/low-resource tasks. We provide a thorough empirical study and analysis of
  this method's benefits showing that it achieves consistent improvements
  relative to the performance trade-off profile of standard static weighting. We
  analyze under what data regimes this method is applicable and show its
  improvements empirically in neural machine translation (NMT) and multi-lingual
  language modeling.
paper_authors._id: 6577efc7dd2996f01ae7077b
paper_authors.name: Dami Choi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-12T08:58:53.917Z
paper_authors._id: 641cacf592cd2530299819aa
paper_authors.avatarUrl: /avatars/7e0ee9d19f05deb320f359b9625a0526.svg
paper_authors.fullname: Dami Choi
paper_authors.type: user
paper_authors.user: choidami
paper_authors._id: 6577efc7dd2996f01ae7077c
paper_authors.name: Derrick Xin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-12T08:59:00.772Z
paper_authors._id: 634cea1605f736dff37abe84
paper_authors.avatarUrl: /avatars/6c199de4b6e7fd1320f47eeecd1310f2.svg
paper_authors.fullname: Derrick Xin
paper_authors.type: user
paper_authors.user: Derr
paper_authors._id: 6577efc7dd2996f01ae7077d
paper_authors.name: Hamid Dadkhahi
paper_authors._id: 6577efc7dd2996f01ae7077e
paper_authors.name: Justin Gilmer
paper_authors._id: 6577efc7dd2996f01ae7077f
paper_authors.name: Ankush Garg
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-12T08:59:12.353Z
paper_authors._id: 609bfde2138207cc2f2f3a8c
paper_authors.avatarUrl: /avatars/8bb0198645bf8aa4e473bab3b1d76b20.svg
paper_authors.fullname: garg
paper_authors.type: user
paper_authors.user: ankushgarg
paper_authors._id: 6577efc7dd2996f01ae70780
paper_authors.name: Orhan Firat
paper_authors._id: 6577efc7dd2996f01ae70781
paper_authors.name: Chih-Kuan Yeh
paper_authors._id: 6577efc7dd2996f01ae70782
paper_authors.name: Andrew M. Dai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-12T08:59:29.785Z
paper_authors._id: 64c04c0b5e03515457ed72a3
paper_authors.avatarUrl: /avatars/dcbfca63b32a8d5c575232316c01dbe7.svg
paper_authors.fullname: Andrew Dai
paper_authors.type: user
paper_authors.user: PenguinPush
paper_authors._id: 6577efc7dd2996f01ae70783
paper_authors.name: Behrooz Ghorbani
paper_title: Order Matters in the Presence of Dataset Imbalance for Multilingual
    Learning
paper_summary: In this paper, we empirically study the optimization dynamics of multi-task
  learning, particularly focusing on those that govern a collection of tasks with
  significant data imbalance. We present a simple yet effective method of
  pre-training on high-resource tasks, followed by fine-tuning on a mixture of
  high/low-resource tasks. We provide a thorough empirical study and analysis of
  this method's benefits showing that it achieves consistent improvements
  relative to the performance trade-off profile of standard static weighting. We
  analyze under what data regimes this method is applicable and show its
  improvements empirically in neural machine translation (NMT) and multi-lingual
  language modeling.
paper_ai_summary: A method of pre-training on high-resource tasks followed by fine-tuning on a mix of high/low-resource tasks improves performance in multi-task learning with data imbalance, particularly in neural machine translation and multi-lingual language modeling.
paper_ai_keywords: multi-task learning data imbalance pre-training fine-tuning static weighting neural machine translation multi-lingual language modeling