title: Cross-Lingual Supervision improves Large Language Models Pre-training
summary: The recent rapid progress in pre-training Large Language Models has relied on
  using self-supervised language modeling objectives like next token prediction
  or span corruption. On the other hand, Machine Translation Systems are mostly
  trained using cross-lingual supervision that requires aligned data between
  source and target languages. We demonstrate that pre-training Large Language
  Models on a mixture of a self-supervised Language Modeling objective and the
  supervised Machine Translation objective, therefore including cross-lingual
  parallel data during pre-training, yields models with better in-context
  learning abilities. As pre-training is a very resource-intensive process and a
  grid search on the best mixing ratio between the two objectives is
  prohibitively expensive, we propose a simple yet effective strategy to learn it
  during pre-training.
paper_authors._id: 646ae99d31fbfc5df8202a47
paper_authors.name: Andrea Schioppa
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-04-23T14:01:37.285Z
paper_authors._id: 6627bf2b8c9f64932fbb13dc
paper_authors.avatarUrl: /avatars/626fa8a5d8e6a9b59d51e580d26f4c3c.svg
paper_authors.fullname: Andrea Schioppa
paper_authors.type: user
paper_authors.user: arischioppa
paper_authors._id: 646ae99d31fbfc5df8202a48
paper_authors.name: Xavier Garcia
paper_authors._id: 646ae99d31fbfc5df8202a49
paper_authors.name: Orhan Firat
paper_title: Cross-Lingual Supervision improves Large Language Models Pre-training
paper_summary: The recent rapid progress in pre-training Large Language Models has relied on
  using self-supervised language modeling objectives like next token prediction
  or span corruption. On the other hand, Machine Translation Systems are mostly
  trained using cross-lingual supervision that requires aligned data between
  source and target languages. We demonstrate that pre-training Large Language
  Models on a mixture of a self-supervised Language Modeling objective and the
  supervised Machine Translation objective, therefore including cross-lingual
  parallel data during pre-training, yields models with better in-context
  learning abilities. As pre-training is a very resource-intensive process and a
  grid search on the best mixing ratio between the two objectives is
  prohibitively expensive, we propose a simple yet effective strategy to learn it
  during pre-training.
paper_ai_summary: Combining self-supervised and supervised objectives during pre-training of Large Language Models improves their in-context learning by incorporating cross-lingual parallel data.
paper_ai_keywords: Large Language Models self-supervised language modeling next token prediction span corruption Machine Translation Systems cross-lingual supervision in-context learning pre-training supervised Machine Translation objective