5

010210321043210443210554321055443210665543210665544321000011011002211022110033221103322110044332211044332211000102103210432105432106543210765432108765432109876543210//2+(4 -4//2 )SoftMaxGrouped AttentionSelfExtendNormal AttentionLLM Maybe LongLM: Self-Extend LLM Context Window Without Tuning

Table 1. Perplexity on PG19 of Llama-2-7b-chat and Mistral-7b-instruct-0.1. Mistral has a unique sliding window attention (SWA)
mechanism at inference. We show the PPL of with&without SWA for Mistral.
Model
Name

Evaluation Context Window Size

12288

10240

14336

16384

8192

4096

6144

Llama-2-7b-chat
SelfExtend-Llama-2-7b-chat
Mistral-7b-instruct-0.1 w/ SWA
Mistral-7b-instruct-0.1 w/o SWA
SelfExtend-Mistral-7b-instruct-0.1

9.181
8.885
9.295
9.295
9.272

> 103
8.828
9.197
9.205
9.103

> 103
9.220
9.532
10.20
9.369

> 103
8.956
9.242
55.35
9.070

> 103
9.217
9.198
> 103
8.956

> 103
9.413
9.278
> 103
9.022

> 103
9.274
9.294
> 103
9.128
Self-Extend LLM