Answers.Body: <p>One prominent reason for preventing large policy updates has to do with the architecture of deep reinforcement learning algorithms: <strong>since parameters are shared by all inputs in a neural network, updating the policy at certain states may also change the policy significantly at other states</strong>. Larger updates generally will result in larger corresponding changes. Once in a while, the change in policy at a state may unintentionally be so large that the algorithm's entire new policy will suffer and yield low expected return.</p>
  <p>The above summary may seem like a slippery slope argument, but <strong>from what I have encountered in practice and from what I've read</strong> (including the two papers you cited), <strong>this decline in performance is a very real phenomenon and happens more often than a practitioner would hope</strong>. For example, there may be a state that has high influence on the expected return of a policy (i.e. a critical state), and an unexpected change of policy at that state may immediately cause performance decline of the algorithm's entire policy until the better policy at that state can be relearned. This performance decline phenomenon can also be intensified by other challenges of reinforcement learning. For example, the credit assignment problem can make it harder to determine which states and actions are contributing to the low expected return, especially in the presence of sparse rewards. Furthermore, exploration hyperparameters may have been annealed before the performance decline took place, making the testing of the unlearned actions less frequent.</p>
  <p>There may be other reasons for the performance decline depending on the environment,  but the reason described above is the one I have most commonly seen in my own work and the literature.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-26T13:35:28.870Z
Answers.Text: I've realized that my assumption of "proper exploration" and "value function that is precise enough" are just not realistic in practical scenarios. There will always be some noise that might **disrupt knowledge that was learnt previously for some other state** which is not part of our current minibatch update. By using updates that somehow guarantee small KL divergence from the previous policy, we're encouraging retention of the knowledge in other states. **However, if we could guarantee that prediction for other states will not be changed, would there be any issue with big policy updates?**
Answers.UserDisplayName: Druudik
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-26T15:23:35.383Z
Answers.Text: @Druudik first of all, thank you for taking the time to write high-quality questions. You are better off writing this comment as another question to get more answers, but I can give a very short sketch of an answer here. One case would be if a good (but not optimal) action at a state (e.g. the start state) is selected frequently during the start of training. If this action generally produces good return, then **it will be chosen more frequently in proportion to the size of the update**. Keeping updates small could help prevent an unwanted/non-optimal feedback loop.
Answers.UserDisplayName: DeepQZero
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-26T03:38:36.613Z
Answers.LastActivityDate: 2024-06-26T03:38:36.613Z
Answers.UserDisplayName: DeepQZero
Body: <p>In the <a href="https://arxiv.org/pdf/1502.05477" rel="nofollow noreferrer">TRPO</a> and <a href="https://arxiv.org/pdf/1707.06347" rel="nofollow noreferrer">PPO</a> papers, it is mentioned that large policy updates often lead to performance drops in policy gradient methods.</p>
  <p>By &quot;<strong>large policy updates</strong>,&quot; they mean a <strong>significant KL divergence</strong> between <strong>subsequent policies</strong>, not necessarily a large difference in the policies' function parameters.</p>
  <p>I have trouble understanding why this would be the case. If we've conducted proper exploration in the current iteration and our value function is precise enough, it seems logical that updating our policy towards more optimal actions should lead to an <strong>improvement</strong>, even if this update is substantial.</p>
  <p>One disadvantage I see is that a <strong>large policy update</strong> could <strong>disrupt our value function</strong>, making the <strong>advantages</strong> in the next iteration <strong>less precise</strong>. However, in this scenario, we could allocate more computational resources to learning a new value function before updating our policy, as in the standard policy evaluation + improvement approach.</p>
  <p>So, my question is: <strong>What are the reasons for preventing big policy updates?</strong></p>
  
Tags: <reinforcement-learning><deep-rl><policy-gradients><function-approximation><policy-iteration>
Title: Why do big policy updates cause performance drop in deep RL?
UserDisplayName: Druudik
ThreadText: # Why do big policy updates cause performance drop in deep RL?
  
  (Asked by: username_0 on 2024-06-25)
  
  In the [TRPO](https://arxiv.org/pdf/1502.05477) and [PPO](https://arxiv.org/pdf/1707.06347) papers, it is mentioned that large policy updates often lead to performance drops in policy gradient methods.
  
  By "**large policy updates**," they mean a **significant KL divergence** between **subsequent policies**, not necessarily a large difference in the policies' function parameters.
  
  I have trouble understanding why this would be the case. If we've conducted proper exploration in the current iteration and our value function is precise enough, it seems logical that updating our policy towards more optimal actions should lead to an **improvement**, even if this update is substantial.
  
  One disadvantage I see is that a **large policy update** could **disrupt our value function**, making the **advantages** in the next iteration **less precise**. However, in this scenario, we could allocate more computational resources to learning a new value function before updating our policy, as in the standard policy evaluation + improvement approach.
  
  So, my question is: **What are the reasons for preventing big policy updates?**
  
  **Tags:** reinforcement-learning, deep-rl, policy-gradients, function-approximation, policy-iteration
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-06-26. Score: 3 | Accepted answer
  
  One prominent reason for preventing large policy updates has to do with the architecture of deep reinforcement learning algorithms: **since parameters are shared by all inputs in a neural network, updating the policy at certain states may also change the policy significantly at other states**. Larger updates generally will result in larger corresponding changes. Once in a while, the change in policy at a state may unintentionally be so large that the algorithm's entire new policy will suffer and yield low expected return.
  
  The above summary may seem like a slippery slope argument, but **from what I have encountered in practice and from what I've read** (including the two papers you cited), **this decline in performance is a very real phenomenon and happens more often than a practitioner would hope**. For example, there may be a state that has high influence on the expected return of a policy (i.e. a critical state), and an unexpected change of policy at that state may immediately cause performance decline of the algorithm's entire policy until the better policy at that state can be relearned. This performance decline phenomenon can also be intensified by other challenges of reinforcement learning. For example, the credit assignment problem can make it harder to determine which states and actions are contributing to the low expected return, especially in the presence of sparse rewards. Furthermore, exploration hyperparameters may have been annealed before the performance decline took place, making the testing of the unlearned actions less frequent.
  
  There may be other reasons for the performance decline depending on the environment, but the reason described above is the one I have most commonly seen in my own work and the literature.
  
  > **Comments:**
  > **username_0**: I've realized that my assumption of "proper exploration" and "value function that is precise enough" are just not realistic in practical scenarios. There will always be some noise that might **disrupt knowledge that was learnt previously for some other state** which is not part of our current minibatch update. By using updates that somehow guarantee small KL divergence from the previous policy, we're encouraging retention of the knowledge in other states. **However, if we could guarantee that prediction for other states will not be changed, would there be any issue with big policy updates?**  
  > **username_1**: @username_0 first of all, thank you for taking the time to write high-quality questions. You are better off writing this comment as another question to get more answers, but I can give a very short sketch of an answer here. One case would be if a good (but not optimal) action at a state (e.g. the start state) is selected frequently during the start of training. If this action generally produces good return, then **it will be chosen more frequently in proportion to the size of the update**. Keeping updates small could help prevent an unwanted/non-optimal feedback loop.  
  
  ---
  
  