Evaluation on these benchmarks is challenging and may be affected by data contamination. We
performed an extensive leaked data analysis after training to ensure the results we report here are as
scientifically sound as possible, but still found some minor issues and decided not to report results on
e.g. LAMBADA (Paperno et al., 2016). As part of the evaluation process, on a popular benchmark,
HellaSwag (Zellers et al., 2019), we find that an additional hundred finetuning steps on specific
website extracts corresponding to the HellaSwag training set (which were not included in Gemini
pretraining set) improve the validation accuracy of Gemini Pro to 89.6% and Gemini Ultra to 96.0%,
when measured with 1-shot prompting (we measured GPT-4 obtained 92.3% when evaluated 1-shot
via the API). This suggests that the benchmark results are susceptible to the pretraining dataset
composition. We choose to report HellaSwag decontaminated results only in a 10-shot evaluation
gemini_1_report