title: Reinforcement Learning Finetunes Small Subnetworks in Large Language
    Models
summary: Reinforcement learning (RL) yields substantial improvements in large language
  models (LLMs) downstream task performance and alignment with human values.
  Surprisingly, such large gains result from updating only a small subnetwork
  comprising just 5 percent to 30 percent of the parameters, with the rest
  effectively unchanged. We refer to this phenomenon as parameter update sparsity
  induced by RL. It is observed across all 7 widely used RL algorithms (e.g.,
  PPO, GRPO, DPO) and all 10 LLMs from different families in our experiments.
  This sparsity is intrinsic and occurs without any explicit sparsity promoting
  regularizations or architectural constraints. Finetuning the subnetwork alone
  recovers the test accuracy, and, remarkably, produces a model nearly identical
  to the one obtained via full finetuning. The subnetworks from different random
  seeds, training data, and even RL algorithms show substantially greater overlap
  than expected by chance. Our analysis suggests that this sparsity is not due to
  updating only a subset of layers, instead, nearly all parameter matrices
  receive similarly sparse updates. Moreover, the updates to almost all parameter
  matrices are nearly full-rank, suggesting RL updates a small subset of
  parameters that nevertheless span almost the full subspaces that the parameter
  matrices can represent. We conjecture that the this update sparsity can be
  primarily attributed to training on data that is near the policy distribution,
  techniques that encourage the policy to remain close to the pretrained model,
  such as the KL regularization and gradient clipping, have limited impact.
paper_authors._id: 682e0d9540c6417d9962227a
paper_authors.name: Sagnik Mukherjee
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-22T07:17:26.467Z
paper_authors._id: 6255a34d7dacca56ac2b04e4
paper_authors.avatarUrl: /avatars/3e7751aa6ef7c880e3e36ac995c9a191.svg
paper_authors.fullname: sagnik mukherjee
paper_authors.type: user
paper_authors.user: sagnikM
paper_authors._id: 682e0d9540c6417d9962227b
paper_authors.name: Lifan Yuan
paper_authors._id: 682e0d9540c6417d9962227c
paper_authors.name: Dilek Hakkani-Tur
paper_authors._id: 682e0d9540c6417d9962227d
paper_authors.name: Hao Peng
paper_title: Reinforcement Learning Finetunes Small Subnetworks in Large Language
    Models
paper_summary: Reinforcement learning (RL) yields substantial improvements in large language
  models (LLMs) downstream task performance and alignment with human values.
  Surprisingly, such large gains result from updating only a small subnetwork
  comprising just 5 percent to 30 percent of the parameters, with the rest
  effectively unchanged. We refer to this phenomenon as parameter update sparsity
  induced by RL. It is observed across all 7 widely used RL algorithms (e.g.,
  PPO, GRPO, DPO) and all 10 LLMs from different families in our experiments.
  This sparsity is intrinsic and occurs without any explicit sparsity promoting
  regularizations or architectural constraints. Finetuning the subnetwork alone
  recovers the test accuracy, and, remarkably, produces a model nearly identical
  to the one obtained via full finetuning. The subnetworks from different random
  seeds, training data, and even RL algorithms show substantially greater overlap
  than expected by chance. Our analysis suggests that this sparsity is not due to
  updating only a subset of layers, instead, nearly all parameter matrices
  receive similarly sparse updates. Moreover, the updates to almost all parameter
  matrices are nearly full-rank, suggesting RL updates a small subset of
  parameters that nevertheless span almost the full subspaces that the parameter
  matrices can represent. We conjecture that the this update sparsity can be
  primarily attributed to training on data that is near the policy distribution,
  techniques that encourage the policy to remain close to the pretrained model,
  such as the KL regularization and gradient clipping, have limited impact.
paper_ai_summary: Reinforcement learning improves large language models with minimal parameter updates, affecting only a small subnetwork without explicit sparsity techniques.
paper_ai_keywords: reinforcement learning large language models parameter update sparsity PPO GRPO DPO policy distribution KL regularization gradient clipping