Searching for Needles in a Haystack: On the Role of Incidental
  Bilingualism in PaLM's Translation Capability
Large, multilingual language models exhibit surprisingly good zero- or
few-shot machine translation capabilities, despite having never seen the
intentionally-included translation examples provided to typical neural
translation systems. We investigate the role of incidental bilingualism -- the
unintentional consumption of bilingual signals, including translation examples
-- in explaining the translation capabilities of large language models, taking
the Pathways Language Model (PaLM) as a case study. We introduce a mixed-method
approach to measure and understand incidental bilingualism at scale. We show
that PaLM is exposed to over 30 million translation pairs across at least 44
languages. Furthermore, the amount of incidental bilingual content is highly
correlated with the amount of monolingual in-language content for non-English
languages. We relate incidental bilingual content to zero-shot prompts and show
that it can be used to mine new prompts to improve PaLM's out-of-English
zero-shot translation quality. Finally, in a series of small-scale ablations,
we show that its presence has a substantial impact on translation capabilities,
although this impact diminishes with model scale.
646587b0e27766e8921baf5f
Eleftheria Briakou
admin_assigned
2023-05-19T14:33:22.355Z
631a06498b8425c8fe66b3b8
/avatars/659e04972fd290fe3eb01e187ffa63d2.svg
Eleftheria Briakou
user
eleftheria
646587b0e27766e8921baf60
Colin Cherry
claimed_verified
2023-05-19T14:33:29.401Z
6144bb22207b8aba44e8ec59
https://cdn-avatars.huggingface.co/v1/production/uploads/1631896530912-6144bb22207b8aba44e8ec59.jpeg
Colin Cherry
user
ColinCherry
646587b0e27766e8921baf61
George Foster
Searching for Needles in a Haystack: On the Role of Incidental
  Bilingualism in PaLM's Translation Capability
Large, multilingual language models exhibit surprisingly good zero- or
few-shot machine translation capabilities, despite having never seen the
intentionally-included translation examples provided to typical neural
translation systems. We investigate the role of incidental bilingualism -- the
unintentional consumption of bilingual signals, including translation examples
-- in explaining the translation capabilities of large language models, taking
the Pathways Language Model (PaLM) as a case study. We introduce a mixed-method
approach to measure and understand incidental bilingualism at scale. We show
that PaLM is exposed to over 30 million translation pairs across at least 44
languages. Furthermore, the amount of incidental bilingual content is highly
correlated with the amount of monolingual in-language content for non-English
languages. We relate incidental bilingual content to zero-shot prompts and show
that it can be used to mine new prompts to improve PaLM's out-of-English
zero-shot translation quality. Finally, in a series of small-scale ablations,
we show that its presence has a substantial impact on translation capabilities,
although this impact diminishes with model scale.
Incidental bilingualism in large multilingual language models, such as PaLM, significantly enhances zero-shot machine translation capabilities by exposure to untranslated translation examples.
zero-shot few-shot machine translation incidental bilingualism bilingual signals translation examples Pathways Language Model PaLM mixed-method translation pairs monolingual in-language content zero-shot prompts small-scale ablations translation capabilities