title: Ignore the KL Penalty! Boosting Exploration on Critical Tokens to
    Enhance RL Fine-Tuning
summary: The ability to achieve long-term goals is a key challenge in the current
  development of large language models (LLMs). To address this, pre-trained LLMs
  can be fine-tuned with reinforcement learning (RL) to explore solutions that
  optimize a given goal. However, exploration with LLMs is difficult, as a
  balance has to be struck between discovering new solutions and staying close
  enough to the pre-trained model, so as not to degrade basic capabilities. This
  is typically controlled with a Kullback-Leibler (KL) penalty. In this paper, we
  investigate the exploration dynamics of a small language model on a simple
  arithmetic task. We show how varying degrees of pre-training influence
  exploration and demonstrate the importance of "critical tokens" which have a
  dramatic impact on the final outcome. Consequently, we introduce a simple
  modification to the KL penalty that favors exploration on critical tokens,
  increasing the efficiency of the RL fine-tuning stage.
paper_authors._id: 67accc647e1fcf03e14b1033
paper_authors.name: Jean Vassoyan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-13T08:24:24.993Z
paper_authors._id: 6637cd3e691043ccb248d0fd
paper_authors.avatarUrl: /avatars/94cf09cf817327be50ecba75f7f60fa1.svg
paper_authors.fullname: Jean Vassoyan
paper_authors.type: user
paper_authors.user: supertardigrade
paper_authors._id: 67accc647e1fcf03e14b1034
paper_authors.name: Nathanaël Beau
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-02-12T16:29:25.829Z
paper_authors._id: 63da60458658cbc1cc489bd7
paper_authors.avatarUrl: /avatars/620ce7ea229de7abe4dc9ea93021f0e4.svg
paper_authors.fullname: Nathanaël Beau
paper_authors.type: user
paper_authors.user: Nbeau
paper_authors._id: 67accc647e1fcf03e14b1035
paper_authors.name: Roman Plaud
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-13T08:24:27.330Z
paper_authors._id: 66470e227d73a39a342866e4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/mDB2nCMAgcX8bQ1u1p7P4.png
paper_authors.fullname: Roman Plaud
paper_authors.type: user
paper_authors.user: lecraquito
paper_title: Ignore the KL Penalty! Boosting Exploration on Critical Tokens to
    Enhance RL Fine-Tuning
paper_summary: The ability to achieve long-term goals is a key challenge in the current
  development of large language models (LLMs). To address this, pre-trained LLMs
  can be fine-tuned with reinforcement learning (RL) to explore solutions that
  optimize a given goal. However, exploration with LLMs is difficult, as a
  balance has to be struck between discovering new solutions and staying close
  enough to the pre-trained model, so as not to degrade basic capabilities. This
  is typically controlled with a Kullback-Leibler (KL) penalty. In this paper, we
  investigate the exploration dynamics of a small language model on a simple
  arithmetic task. We show how varying degrees of pre-training influence
  exploration and demonstrate the importance of "critical tokens" which have a
  dramatic impact on the final outcome. Consequently, we introduce a simple
  modification to the KL penalty that favors exploration on critical tokens,
  increasing the efficiency of the RL fine-tuning stage.
paper_ai_summary: A modification to the KL penalty in RL fine-tuning of language models enhances exploration, particularly on critical tokens, improving the efficiency of reaching long-term goals.
paper_ai_keywords: reinforcement learning pre-trained LLMs KL penalty exploration critical tokens