Gemini Ultra also excels in coding, a popular use case of current LLMs. We evaluate the model
on many conventional and internal benchmarks and also measure its performance as part of more
complex reasoning systems such as AlphaCode 2 (see section 5.1.7 on complex reasoning systems).
For example, on HumanEval, a standard code-completion benchmark (Chen et al., 2021) mapping
function descriptions to Python implementations, instruction-tuned Gemini Ultra correctly implements
74.4% of problems. On a new held-out evaluation benchmark for python code generation tasks,
Natural2Code, where we ensure no web leakage, Gemini Ultra achieves the highest score of 74.9%.

6

Gemini: A Family of Highly Capable Multimodal Models

GPT-4

GPT-3.5

PaLM 2-L

Claude 2

Grok 1

LLAMA-2

70%
5-shot

78.4%
5-shot

78.5%
5-shot CoT

73.0%
5-shot

68.0%∗∗∗

Gemini
Ultra
90.04%
CoT@32∗

83.7%
5-shot
94.4%
Maj1@32
53.2%
4-shot

Gemini
Pro
79.13%
CoT@8∗

71.8%
5-shot

86.5%
Maj1@32

32.6%
4-shot

83.6%
3-shot
gemini_1_report