Most existing works of context length extension rely on lan-
guage modeling (measured by PPL) and synthetic tasks such
as passkey retrieval to measure LLMs’ real long context ca-
pabilities. However, such tasks cannot comprehensively
reflect LLMs’ long context capabilities. Passkey retrieval is
too easy and LLM may not be able to handle long context
well with low PPL.
To gauge long-context performance, we perform the evalu-
ation using two real-world long context evaluation bench-
marks: Longbench (Bai et al., 2023) and L-Eval (An et al.,
2023). The results are listed in Table 2 and Table 3 Some re-
sults in these tables are reported from other literature, which
is indicated by footnotes.
On the Longbench, for all
three different base mod-
els and most datasets, with Self-Extend applied, com-
pared to the counter part, the model can obtain sig-
nificant performance boost (SEext-Llama-2-7B-chat vs.
Llama-2-7B-chat; SExt-Mistral-7B-ins-0.1 vs. Mistral-
Self-Extend LLM