Variance Control via Weight Rescaling in LLM Pre-training
The outcome of Large Language Model (LLM) pre-training strongly depends on
weight initialization and variance control strategies. Although the importance
of initial variance control has been well documented in neural networks in
general, the literature on initialization and management of its growth during
LLM pre-training, specifically, is somewhat sparse. In this paper, we introduce
the Layer Index Rescaling (LIR) weight initialization scheme, and the Target
Variance Rescaling (TVR) variance control strategy. Experiments on a 1B
parameter LLaMA model demonstrate that better variance management using these
techniques yields substantial improvements in downstream task performance (up
to 4.6% on common pre-training benchmarks) and reduces extreme activation
values, thus mitigating challenges associated with quantization and
low-precision training. Our code is available at:
https://github.com/bluorion-com/weight_rescaling.
67e23d503ef5318b1550f1bc
Louis Owen
claimed_verified
2025-03-25T08:19:59.211Z
6071c4b270e11b30cfcfd7a3
https://cdn-avatars.huggingface.co/v1/production/uploads/6071c4b270e11b30cfcfd7a3/-1ekCBzSTpqxkkul0bgmI.jpeg
Louis Owen
user
louisowen6
67e23d503ef5318b1550f1bd
Abhay Kumar
claimed_verified
2025-03-25T09:23:34.033Z
62cd4b03c5cc157be82f0b56
/avatars/351e963c1c763d507ae78cbcd62966a3.svg
Abhay kumar
user
akanyaani
67e23d503ef5318b1550f1be
Nilabhra Roy Chowdhury
claimed_verified
2025-03-25T09:15:43.961Z
645a0d3dd6648853107c5fdc
/avatars/1e3b6a4f5ce81a707ba7cbdf81631091.svg
Nilabhra Roy Chowdhury
user
nilabhra
67e23d503ef5318b1550f1bf
Fabian Güra
claimed_verified
2025-03-25T08:19:56.909Z
65e4be59e8b017ee1310a1b6
/avatars/c3f7cdf5d0859cb80bfb2b970a675dfa.svg
Fabian
user
gueraf
Variance Control via Weight Rescaling in LLM Pre-training
The outcome of Large Language Model (LLM) pre-training strongly depends on
weight initialization and variance control strategies. Although the importance
of initial variance control has been well documented in neural networks in
general, the literature on initialization and management of its growth during
LLM pre-training, specifically, is somewhat sparse. In this paper, we introduce
the Layer Index Rescaling (LIR) weight initialization scheme, and the Target
Variance Rescaling (TVR) variance control strategy. Experiments on a 1B
parameter LLaMA model demonstrate that better variance management using these
techniques yields substantial improvements in downstream task performance (up
to 4.6% on common pre-training benchmarks) and reduces extreme activation
values, thus mitigating challenges associated with quantization and
low-precision training. Our code is available at:
https://github.com/bluorion-com/weight_rescaling.
The Layer Index Rescaling (LIR) and Target Variance Rescaling (TVR) techniques improve variance management during LLM pre-training, leading to better performance and mitigating quantization and low-precision training challenges.
Large Language Model (LLM) weight initialization variance control Layer Index Rescaling (LIR) Target Variance Rescaling (TVR) downstream task performance quantization low-precision training