title: Unsupervised Neural Machine Translation with Generative Language Models Only
abstract: We show how to derive state-of-the-art unsupervised neural machine translation systems from generatively pre-trained language models. Our method consists of three steps: \emph{few-shot amplification}, \emph{distillation}, and \emph{backtranslation}. We first use the zero-shot translation ability of large pretrained language models to generate translations for a small set of unlabeled sentences.  We then amplify these zero-shot translations by using them as few-shot demonstrations for sampling a larger synthetic dataset. This dataset is then distilled by discarding the few-shot demonstrations and then fine-tuning. During backtranslation, we repeatedly generate translations for a set of inputs and then fine-tune a single language model on both directions of the translation task at once, ensuring cycle-consistency by swapping the roles of gold monotext and generated translations when fine-tuning. By using our method to leverage GPT-3's zero-shot translation capability, we achieve a new state-of-the-art in unsupervised translation on the WMT14 English-French benchmark, attaining a BLEU score of 42.1.  
authors: Jesse Michael Han Igor Babuschkin Harrison Edwards Arvind Neelakantan Tao Xu Stanislas Polu Alex Ray Pranav Shyam Aditya Ramesh Alec Radford Ilya Sutskever
hIndex: Ilya Sutskever 63