title: Variance Control via Weight Rescaling in LLM Pre-training
summary: The outcome of Large Language Model (LLM) pre-training strongly depends on
  weight initialization and variance control strategies. Although the importance
  of initial variance control has been well documented in neural networks in
  general, the literature on initialization and management of its growth during
  LLM pre-training, specifically, is somewhat sparse. In this paper, we introduce
  the Layer Index Rescaling (LIR) weight initialization scheme, and the Target
  Variance Rescaling (TVR) variance control strategy. Experiments on a 1B
  parameter LLaMA model demonstrate that better variance management using these
  techniques yields substantial improvements in downstream task performance (up
  to 4.6% on common pre-training benchmarks) and reduces extreme activation
  values, thus mitigating challenges associated with quantization and
  low-precision training. Our code is available at:
  https://github.com/bluorion-com/weight_rescaling.
paper_authors._id: 67e23d503ef5318b1550f1bc
paper_authors.name: Louis Owen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-25T08:19:59.211Z
paper_authors._id: 6071c4b270e11b30cfcfd7a3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6071c4b270e11b30cfcfd7a3/-1ekCBzSTpqxkkul0bgmI.jpeg
paper_authors.fullname: Louis Owen
paper_authors.type: user
paper_authors.user: louisowen6
paper_authors._id: 67e23d503ef5318b1550f1bd
paper_authors.name: Abhay Kumar
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-25T09:23:34.033Z
paper_authors._id: 62cd4b03c5cc157be82f0b56
paper_authors.avatarUrl: /avatars/351e963c1c763d507ae78cbcd62966a3.svg
paper_authors.fullname: Abhay kumar
paper_authors.type: user
paper_authors.user: akanyaani
paper_authors._id: 67e23d503ef5318b1550f1be
paper_authors.name: Nilabhra Roy Chowdhury
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-25T09:15:43.961Z
paper_authors._id: 645a0d3dd6648853107c5fdc
paper_authors.avatarUrl: /avatars/1e3b6a4f5ce81a707ba7cbdf81631091.svg
paper_authors.fullname: Nilabhra Roy Chowdhury
paper_authors.type: user
paper_authors.user: nilabhra
paper_authors._id: 67e23d503ef5318b1550f1bf
paper_authors.name: Fabian Güra
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-25T08:19:56.909Z
paper_authors._id: 65e4be59e8b017ee1310a1b6
paper_authors.avatarUrl: /avatars/c3f7cdf5d0859cb80bfb2b970a675dfa.svg
paper_authors.fullname: Fabian
paper_authors.type: user
paper_authors.user: gueraf
paper_title: Variance Control via Weight Rescaling in LLM Pre-training
paper_summary: The outcome of Large Language Model (LLM) pre-training strongly depends on
  weight initialization and variance control strategies. Although the importance
  of initial variance control has been well documented in neural networks in
  general, the literature on initialization and management of its growth during
  LLM pre-training, specifically, is somewhat sparse. In this paper, we introduce
  the Layer Index Rescaling (LIR) weight initialization scheme, and the Target
  Variance Rescaling (TVR) variance control strategy. Experiments on a 1B
  parameter LLaMA model demonstrate that better variance management using these
  techniques yields substantial improvements in downstream task performance (up
  to 4.6% on common pre-training benchmarks) and reduces extreme activation
  values, thus mitigating challenges associated with quantization and
  low-precision training. Our code is available at:
  https://github.com/bluorion-com/weight_rescaling.
paper_ai_summary: The Layer Index Rescaling (LIR) and Target Variance Rescaling (TVR) techniques improve variance management during LLM pre-training, leading to better performance and mitigating quantization and low-precision training challenges.
paper_ai_keywords: Large Language Model (LLM) weight initialization variance control Layer Index Rescaling (LIR) Target Variance Rescaling (TVR) downstream task performance quantization low-precision training