E^2-LLM: Efficient and Extreme Length Extension of Large Language Models
Typically, training LLMs with long context sizes is computationally
expensive, requiring extensive training hours and GPU resources. Existing
long-context extension methods usually need additional training procedures to
support corresponding long-context windows, where the long-context training
data (e.g., 32k) is needed, and high GPU training costs are assumed. To address
the aforementioned issues, we propose an Efficient and Extreme length extension
method for Large Language Models, called E 2 -LLM, with only one training
procedure and dramatically reduced computation cost, which also removes the
need to collect long-context data. Concretely, first, the training data of our
E 2 -LLM only requires a short length (e.g., 4k), which reduces the tuning cost
greatly. Second, the training procedure on the short training context window is
performed only once time, and we can support different evaluation context
windows at inference. Third, in E 2 - LLM, based on RoPE position embeddings,
we introduce two different augmentation methods on the scale and position index
parameters for different samples in training. It aims to make the model more
robust to the different relative differences when directly interpolating the
arbitrary context length at inference. Comprehensive experimental results on
multiple benchmark datasets demonstrate the effectiveness of our E 2 -LLM on
challenging long-context tasks.
65a77b83e94d40886a883bd3
Jiaheng Liu
claimed_verified
2024-01-17T13:44:48.548Z
65377c30e48353201e6fdda0
/avatars/a8f803b6f2e598eaee9c52c0d2ddfc16.svg
Jiaheng Liu
user
CheeryLJH
65a77b83e94d40886a883bd4
Zhiqi Bai
claimed_verified
2024-01-18T08:20:41.032Z
650d40a8747f5328a2253dbc
/avatars/5d9186658fb8ab853dff6d2d75d03efb.svg
Zhiqi Bai
user
bzq
65a77b83e94d40886a883bd5
Yuanxing Zhang
65a77b83e94d40886a883bd6
Chenchen Zhang
claimed_verified
2024-06-02T18:48:31.688Z
64b74b906ab5d14ca7f289cd
/avatars/b131b7c4ce5216708ca4a678f35ead0a.svg
xxzcc
user
xxzcc
65a77b83e94d40886a883bd7
Yu Zhang
65a77b83e94d40886a883bd8
Ge Zhang
admin_assigned
2024-01-17T09:57:41.725Z
638efcf4c67af472d316d424
/avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
Ge Zhang
user
zhangysk
65a77b83e94d40886a883bd9
Jiakai Wang
65a77b83e94d40886a883bda
Haoran Que
65a77b83e94d40886a883bdb
Yukang Chen
admin_assigned
2024-01-17T09:58:28.966Z
62919485a29097b211bc7b83
https://cdn-avatars.huggingface.co/v1/production/uploads/1653710384819-62919485a29097b211bc7b83.png
YukangChen
user
Yukang
65a77b83e94d40886a883bdc
Wenbo Su
65a77b83e94d40886a883bdd
Tiezheng Ge
65a77b83e94d40886a883bde
Jie Fu
admin_assigned
2024-01-17T09:59:26.227Z
641a6895fb5ffff5ac79d593
https://cdn-avatars.huggingface.co/v1/production/uploads/641a6895fb5ffff5ac79d593/vxvwsto3llOEWGqQKGMYx.jpeg
Jie Fu
user
bigaidream
65a77b83e94d40886a883bdf
Wenhu Chen
admin_assigned
2024-01-17T09:59:32.941Z
6313a86154e6e5d9f0f94e04
https://cdn-avatars.huggingface.co/v1/production/uploads/1662232951344-6313a86154e6e5d9f0f94e04.jpeg
Wenhu Chen
user
wenhu
65a77b83e94d40886a883be0
Bo Zheng
E^2-LLM: Efficient and Extreme Length Extension of Large Language Models
Typically, training LLMs with long context sizes is computationally
expensive, requiring extensive training hours and GPU resources. Existing
long-context extension methods usually need additional training procedures to
support corresponding long-context windows, where the long-context training
data (e.g., 32k) is needed, and high GPU training costs are assumed. To address
the aforementioned issues, we propose an Efficient and Extreme length extension
method for Large Language Models, called E 2 -LLM, with only one training
procedure and dramatically reduced computation cost, which also removes the
need to collect long-context data. Concretely, first, the training data of our
E 2 -LLM only requires a short length (e.g., 4k), which reduces the tuning cost
greatly. Second, the training procedure on the short training context window is
performed only once time, and we can support different evaluation context
windows at inference. Third, in E 2 - LLM, based on RoPE position embeddings,
we introduce two different augmentation methods on the scale and position index
parameters for different samples in training. It aims to make the model more
robust to the different relative differences when directly interpolating the
arbitrary context length at inference. Comprehensive experimental results on
multiple benchmark datasets demonstrate the effectiveness of our E 2 -LLM on
challenging long-context tasks.
E 2 -LLM extends long-context capabilities in LLMs with reduced computation and data requirements through single training, short context windows, and novel position embedding augmentations.
LLMs long context GPU resources long-context extension methods Efficient and Extreme length extension E 2 -LLM short length RoPE position embeddings augmentation methods scale and position index parameters model robustness relative differences interpolation benchmark datasets