74.7
15.4
50.4

Table 5 | Performance of Gemini models on multilingual math and summarization.

5.1.5. Long Context
Gemini models are trained with a sequence length of 32,768 tokens and we find that they make use
of their context length effectively. We first verify this by running a synthetic retrieval test: we place
key-value pairs at the beginning of the context, then add long filler text, and ask for value associated
with a particular key. We find that the Ultra model retrieves the correct value with 98% accuracy
when queried across the full context length. We further investigate this by plotting the negative log
likelihood (NLL) versus the token index across a held-out set of long documents in Figure 4. We
find that the NLL decreases with sequence position up to the full 32K context length. The longer
context length of Gemini models enable new use cases such as retrieval over documents and video
understanding discussed in section 5.2.2.
gemini_1_report