write using relatively short texts, such as articles spanning
several pages. We rarely use extremely long texts like entire
books or complete documents as learning materials. Yet,
we are still able to understand long texts effectively. With
this strong motivation, the poor performance of LLMs while
facing long text out of the pretraining context window size is
not due to the lack of long context understanding capabilities.
We suppose that there should be a way to elicit LLMs’
inherent long context capability.
In our analysis, we observe that the key challenge prevent-
ing LLMs from effectively managing extensive contexts is
Self-Extend LLM