temperature sampling with τ = 0.7 to generate ﬁve responses randomly, and then rank them by log probability
score without length normalization. We choose the response with the best score, after a ﬁltering step of
removing any generations with scores that were better than half of the median score, which we found
successfully removed a large portion of generations with undesirable repetitions. For example, if the median
log probability score of ﬁve generations is -20, then a generation with a score of -3 would likely have
undesirable repetitions and we ﬁlter it out. We then present the PaLM and Flan-PaLM outputs to human
raters and ask them to choose the responses based on desirability. Each pair of outputs is scored by one rater.
The human rater instructions are given in Appendix I.
The results of this human evaluation are
shown in Figure 8—across 190 examples,
Flan-PaLM generations were preferred 79%
of the time. For every zero-shot setting, Flan-
Scaling Instruction-Finetuned Language Models