7

LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuning

Table 3. Exam evaluation results on some closed-ended tasks from L-Eval. Tokens denotes the maximum input context length. +
indicates the results are from us and others are reported by L-Eval. The rows in the same color (orange, green, blue, and pink) represent
the models of those rows from the same base model. The best performance is in bold and the second best is underlined
SFiction

QuALITY

Coursera

Model
Claude1.3-100k
GPT-4-32k
Turbo-16k-0613
Chatglm2-6b-8k
XGen-7b-8k (2k-4k-8k)
Chatglm2-6b-8k
Chatglm2-6b-32k
XGen-7b-8k
MPT-7b-65k
Llama2-7b-chat
Longchat1.5-7b-32k
Llama2-7b-NTK
SelfExtend-Llama2-7B-chat+
Vicuna1.5-7b-16k
SelfExtend-Vicuna1.5-7B+
Llama2-13b-chat
Llama2-13b-NTK
Llama2-13b-NTK(Dyn)
SelfExtend-Llama2-13B-chat+
Mistral-7b-ins-0.1 w/ SWA+
MistralLite+
SelfExtend-Mistral-7b-ins-0.1+
SOLAR-10.7b-Instruct-v1.0+
SEext-SOLAR-10.7b-Instruct-v1.0+
Self-Extend LLM