title: E^2-LLM: Efficient and Extreme Length Extension of Large Language Models
summary: Typically, training LLMs with long context sizes is computationally
  expensive, requiring extensive training hours and GPU resources. Existing
  long-context extension methods usually need additional training procedures to
  support corresponding long-context windows, where the long-context training
  data (e.g., 32k) is needed, and high GPU training costs are assumed. To address
  the aforementioned issues, we propose an Efficient and Extreme length extension
  method for Large Language Models, called E 2 -LLM, with only one training
  procedure and dramatically reduced computation cost, which also removes the
  need to collect long-context data. Concretely, first, the training data of our
  E 2 -LLM only requires a short length (e.g., 4k), which reduces the tuning cost
  greatly. Second, the training procedure on the short training context window is
  performed only once time, and we can support different evaluation context
  windows at inference. Third, in E 2 - LLM, based on RoPE position embeddings,
  we introduce two different augmentation methods on the scale and position index
  parameters for different samples in training. It aims to make the model more
  robust to the different relative differences when directly interpolating the
  arbitrary context length at inference. Comprehensive experimental results on
  multiple benchmark datasets demonstrate the effectiveness of our E 2 -LLM on
  challenging long-context tasks.
paper_authors._id: 65a77b83e94d40886a883bd3
paper_authors.name: Jiaheng Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-17T13:44:48.548Z
paper_authors._id: 65377c30e48353201e6fdda0
paper_authors.avatarUrl: /avatars/a8f803b6f2e598eaee9c52c0d2ddfc16.svg
paper_authors.fullname: Jiaheng Liu
paper_authors.type: user
paper_authors.user: CheeryLJH
paper_authors._id: 65a77b83e94d40886a883bd4
paper_authors.name: Zhiqi Bai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-18T08:20:41.032Z
paper_authors._id: 650d40a8747f5328a2253dbc
paper_authors.avatarUrl: /avatars/5d9186658fb8ab853dff6d2d75d03efb.svg
paper_authors.fullname: Zhiqi Bai
paper_authors.type: user
paper_authors.user: bzq
paper_authors._id: 65a77b83e94d40886a883bd5
paper_authors.name: Yuanxing Zhang
paper_authors._id: 65a77b83e94d40886a883bd6
paper_authors.name: Chenchen Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-02T18:48:31.688Z
paper_authors._id: 64b74b906ab5d14ca7f289cd
paper_authors.avatarUrl: /avatars/b131b7c4ce5216708ca4a678f35ead0a.svg
paper_authors.fullname: xxzcc
paper_authors.type: user
paper_authors.user: xxzcc
paper_authors._id: 65a77b83e94d40886a883bd7
paper_authors.name: Yu Zhang
paper_authors._id: 65a77b83e94d40886a883bd8
paper_authors.name: Ge Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-17T09:57:41.725Z
paper_authors._id: 638efcf4c67af472d316d424
paper_authors.avatarUrl: /avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
paper_authors.fullname: Ge Zhang
paper_authors.type: user
paper_authors.user: zhangysk
paper_authors._id: 65a77b83e94d40886a883bd9
paper_authors.name: Jiakai Wang
paper_authors._id: 65a77b83e94d40886a883bda
paper_authors.name: Haoran Que
paper_authors._id: 65a77b83e94d40886a883bdb
paper_authors.name: Yukang Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-17T09:58:28.966Z
paper_authors._id: 62919485a29097b211bc7b83
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1653710384819-62919485a29097b211bc7b83.png
paper_authors.fullname: YukangChen
paper_authors.type: user
paper_authors.user: Yukang
paper_authors._id: 65a77b83e94d40886a883bdc
paper_authors.name: Wenbo Su
paper_authors._id: 65a77b83e94d40886a883bdd
paper_authors.name: Tiezheng Ge
paper_authors._id: 65a77b83e94d40886a883bde
paper_authors.name: Jie Fu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-17T09:59:26.227Z
paper_authors._id: 641a6895fb5ffff5ac79d593
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/641a6895fb5ffff5ac79d593/dFR_ofjbqCrcqGa9R3MMq.jpeg
paper_authors.fullname: Jie Fu
paper_authors.type: user
paper_authors.user: bigaidream
paper_authors._id: 65a77b83e94d40886a883bdf
paper_authors.name: Wenhu Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-17T09:59:32.941Z
paper_authors._id: 6313a86154e6e5d9f0f94e04
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1662232951344-6313a86154e6e5d9f0f94e04.jpeg
paper_authors.fullname: Wenhu Chen
paper_authors.type: user
paper_authors.user: wenhu
paper_authors._id: 65a77b83e94d40886a883be0
paper_authors.name: Bo Zheng
paper_title: E^2-LLM: Efficient and Extreme Length Extension of Large Language Models
paper_summary: Typically, training LLMs with long context sizes is computationally
  expensive, requiring extensive training hours and GPU resources. Existing
  long-context extension methods usually need additional training procedures to
  support corresponding long-context windows, where the long-context training
  data (e.g., 32k) is needed, and high GPU training costs are assumed. To address
  the aforementioned issues, we propose an Efficient and Extreme length extension
  method for Large Language Models, called E 2 -LLM, with only one training
  procedure and dramatically reduced computation cost, which also removes the
  need to collect long-context data. Concretely, first, the training data of our
  E 2 -LLM only requires a short length (e.g., 4k), which reduces the tuning cost
  greatly. Second, the training procedure on the short training context window is
  performed only once time, and we can support different evaluation context
  windows at inference. Third, in E 2 - LLM, based on RoPE position embeddings,
  we introduce two different augmentation methods on the scale and position index
  parameters for different samples in training. It aims to make the model more
  robust to the different relative differences when directly interpolating the
  arbitrary context length at inference. Comprehensive experimental results on
  multiple benchmark datasets demonstrate the effectiveness of our E 2 -LLM on
  challenging long-context tasks.
paper_ai_keywords: Large Language Models (LLMs) long context sizes GPU resources long-context extension methods long-context training data Efficient and Extreme length extension method for Large Language Models (E 2 -LLM) training data short length training procedure inference context windows RoPE position embeddings augmentation methods scale and position index parameters inference