title: Searching for Needles in a Haystack: On the Role of Incidental
    Bilingualism in PaLM's Translation Capability
summary: Large, multilingual language models exhibit surprisingly good zero- or
  few-shot machine translation capabilities, despite having never seen the
  intentionally-included translation examples provided to typical neural
  translation systems. We investigate the role of incidental bilingualism -- the
  unintentional consumption of bilingual signals, including translation examples
  -- in explaining the translation capabilities of large language models, taking
  the Pathways Language Model (PaLM) as a case study. We introduce a mixed-method
  approach to measure and understand incidental bilingualism at scale. We show
  that PaLM is exposed to over 30 million translation pairs across at least 44
  languages. Furthermore, the amount of incidental bilingual content is highly
  correlated with the amount of monolingual in-language content for non-English
  languages. We relate incidental bilingual content to zero-shot prompts and show
  that it can be used to mine new prompts to improve PaLM's out-of-English
  zero-shot translation quality. Finally, in a series of small-scale ablations,
  we show that its presence has a substantial impact on translation capabilities,
  although this impact diminishes with model scale.
paper_authors._id: 646587b0e27766e8921baf5f
paper_authors.name: Eleftheria Briakou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-19T14:33:22.355Z
paper_authors._id: 631a06498b8425c8fe66b3b8
paper_authors.avatarUrl: /avatars/659e04972fd290fe3eb01e187ffa63d2.svg
paper_authors.fullname: Eleftheria Briakou
paper_authors.type: user
paper_authors.user: eleftheria
paper_authors._id: 646587b0e27766e8921baf60
paper_authors.name: Colin Cherry
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-19T14:33:29.401Z
paper_authors._id: 6144bb22207b8aba44e8ec59
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1631896530912-6144bb22207b8aba44e8ec59.jpeg
paper_authors.fullname: Colin Cherry
paper_authors.type: user
paper_authors.user: ColinCherry
paper_authors._id: 646587b0e27766e8921baf61
paper_authors.name: George Foster
paper_title: Searching for Needles in a Haystack: On the Role of Incidental
    Bilingualism in PaLM's Translation Capability
paper_summary: Large, multilingual language models exhibit surprisingly good zero- or
  few-shot machine translation capabilities, despite having never seen the
  intentionally-included translation examples provided to typical neural
  translation systems. We investigate the role of incidental bilingualism -- the
  unintentional consumption of bilingual signals, including translation examples
  -- in explaining the translation capabilities of large language models, taking
  the Pathways Language Model (PaLM) as a case study. We introduce a mixed-method
  approach to measure and understand incidental bilingualism at scale. We show
  that PaLM is exposed to over 30 million translation pairs across at least 44
  languages. Furthermore, the amount of incidental bilingual content is highly
  correlated with the amount of monolingual in-language content for non-English
  languages. We relate incidental bilingual content to zero-shot prompts and show
  that it can be used to mine new prompts to improve PaLM's out-of-English
  zero-shot translation quality. Finally, in a series of small-scale ablations,
  we show that its presence has a substantial impact on translation capabilities,
  although this impact diminishes with model scale.
paper_ai_keywords: large, multilingual language models zero-shot machine translation few-shot machine translation incidental bilingualism bilingual signals translation pairs Pathways Language Model (PaLM) mixed approach monolingual in-language content zero-shot prompts out-of-English zero-shot translation quality ablations