01234Number of Shots0%20%40%60%80%100%AccuracyHuman multiple-choice baselineFlan-Palm-540bFlan-Palm-62bFlan-Palm-8bFlan-T5-xxl-11bPalm-540bPalm-62bPalm-8b0.00.20.40.60.81.0Accuracy0 shot1 shotFlan4 shot81162535Parameters (billion)0.00.20.40.60.81.0Accuracy81162535Parameters (billion)81162535Parameters (billion)PaLMgotchastereotypicalC.5 Translation misgendering
We additionally measure the impact of instruction ﬁnetuning on an evaluation measuring potential misgen-
dering harms when translating sentences that encode gender unambiguously. We evaluate translation into
English from 26 source languages at diﬀerent resources levels (Appendix C.5), including very low resource
languages that are underrepresented in digital spaces (Bapna et al., 2022). Evaluation sets are constructed so
that the source language input contains only a single gendered entity, which enables automated scoring of
Scaling Instruction-Finetuned Language Models