4.1. Performance on language modeling

Language modeling is the most fundamental and the least
requirement to a LLM. A low PPL does not guarantee good
performance on real tasks while a too high PPL suggests
severe performance degradation of LLMs.
We evaluate Self-Extend’s language modeling performance
on PG19 (Rae et al., 2019), which contains long books.
We use the first sentence of each book in PG19’s test
Self-Extend LLM