title: CLEX: Continuous Length Extrapolation for Large Language Models
summary: Transformer-based Large Language Models (LLMs) are pioneering advances in
  many natural language processing tasks, however, their exceptional capabilities
  are restricted within the preset context window of Transformer. Position
  Embedding (PE) scaling methods, while effective in extending the context window
  to a specific length, demonstrate either notable limitations in their
  extrapolation abilities or sacrificing partial performance within the context
  window. Length extrapolation methods, although theoretically capable of
  extending the context window beyond the training sequence length, often
  underperform in practical long-context applications. To address these
  challenges, we propose Continuous Length EXtrapolation (CLEX) for LLMs. We
  generalise the PE scaling approaches to model the continuous dynamics by
  ordinary differential equations over the length scaling factor, thereby
  overcoming the constraints of current PE scaling methods designed for specific
  lengths. Moreover, by extending the dynamics to desired context lengths beyond
  the training sequence length, CLEX facilitates the length extrapolation with
  impressive performance in practical tasks. We demonstrate that CLEX can be
  seamlessly incorporated into LLMs equipped with Rotary Position Embedding, such
  as LLaMA and GPT-NeoX, with negligible impact on training and inference
  latency. Experimental results reveal that CLEX can effectively extend the
  context window to over 4x or almost 8x training length, with no deterioration
  in performance. Furthermore, when evaluated on the practical LongBench
  benchmark, our model trained on a 4k length exhibits competitive performance
  against state-of-the-art open-source models trained on context lengths up to
  32k.
paper_authors._id: 6539e1ead1ca3238d0e72c42
paper_authors.name: Guanzheng Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-26T08:15:50.170Z
paper_authors._id: 645475e2548f22be59847604
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/645475e2548f22be59847604/EhSurrZ25u31qQ2TVXQXt.jpeg
paper_authors.fullname: Chen
paper_authors.type: user
paper_authors.user: Guanzheng
paper_authors._id: 6539e1ead1ca3238d0e72c43
paper_authors.name: Xin Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-26T07:32:27.706Z
paper_authors._id: 63913b120cf6b11c487ca31d
paper_authors.avatarUrl: /avatars/aec44edd5470dd6e767e0a25efd6fb5d.svg
paper_authors.fullname: Xin Li
paper_authors.type: user
paper_authors.user: lixin4ever
paper_authors._id: 6539e1ead1ca3238d0e72c44
paper_authors.name: Zaiqiao Meng
paper_authors._id: 6539e1ead1ca3238d0e72c45
paper_authors.name: Shangsong Liang
paper_authors._id: 6539e1ead1ca3238d0e72c46
paper_authors.name: Lidong Bing
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-26T08:16:27.010Z
paper_authors._id: 6454685a548f22be598414c4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/eMjMWKJ-AouF7eY1-RzGF.jpeg
paper_authors.fullname: Lidong Bing
paper_authors.type: user
paper_authors.user: LidongBing
paper_title: CLEX: Continuous Length Extrapolation for Large Language Models
paper_summary: Transformer-based Large Language Models (LLMs) are pioneering advances in
  many natural language processing tasks, however, their exceptional capabilities
  are restricted within the preset context window of Transformer. Position
  Embedding (PE) scaling methods, while effective in extending the context window
  to a specific length, demonstrate either notable limitations in their
  extrapolation abilities or sacrificing partial performance within the context
  window. Length extrapolation methods, although theoretically capable of
  extending the context window beyond the training sequence length, often
  underperform in practical long-context applications. To address these
  challenges, we propose Continuous Length EXtrapolation (CLEX) for LLMs. We
  generalise the PE scaling approaches to model the continuous dynamics by
  ordinary differential equations over the length scaling factor, thereby
  overcoming the constraints of current PE scaling methods designed for specific
  lengths. Moreover, by extending the dynamics to desired context lengths beyond
  the training sequence length, CLEX facilitates the length extrapolation with
  impressive performance in practical tasks. We demonstrate that CLEX can be
  seamlessly incorporated into LLMs equipped with Rotary Position Embedding, such
  as LLaMA and GPT-NeoX, with negligible impact on training and inference
  latency. Experimental results reveal that CLEX can effectively extend the
  context window to over 4x or almost 8x training length, with no deterioration
  in performance. Furthermore, when evaluated on the practical LongBench
  benchmark, our model trained on a 4k length exhibits competitive performance
  against state-of-the-art open-source models trained on context lengths up to
  32k.
paper_ai_keywords: Transformer-based Large Language Models (LLMs) Position Embedding (PE) scaling methods Length extrapolation methods Continuous Length EXtrapolation (CLEX) ordinary differential equations Rotary Position Embedding LongBench benchmark