4.2 Long Contexts

Earlier this year, we expanded Claude’s context window from 9K to 100K tokens. Claude 2 has been trained
to have a further expanded context window of 200K tokens, corresponding to roughly 150,000 words. To
demonstrate that Claude is actually using the full context, we measure the loss for each token position,
averaged over 1000 long documents, in Figure 8. The per-token loss has a power-law plus constant trend, as
expected based on [21].
As we note in our launch blog post, we will support 100K at launch rather than this full context window.
However, we may integrate this underlying capability into our product offering at a later date.

Figure 8 This figure shows the loss as a function of token position for Claude 2 on very long context data, along with a
fit to a power-law plus constant function. These results demonstrate that Claude 2 continues to show gains in performance
(on the autoregressive cross-entropy loss) up to 200k tokens of text.
ClaudeModels