Answers.Body: <blockquote>
    <p>Why do RL implementations converge on one action?</p>
  </blockquote>
  
  <p>If the optimal policy shouldn't always select the same action in the same state, i.e., if the optimal policy isn't <a href="https://ai.stackexchange.com/a/12275/2444">deterministic</a> (e.g., in the case of the <em>rock paper scissors</em>, the optimal policy cannot be deterministic because any intelligent player would easily memorize your deterministic policy, so, after a while, you would always lose again that player), then there are a few things that you can do to make your policy more stochastic</p>
  
  <ol>
  <li><p>Change the reward function. If your agent ends up selecting always the same action and you don't want that, it's probably because you're not giving it the right reinforcement signal (given that the agent selects the action that apparently will give it the highest reward in the long run).</p></li>
  <li><p>Try to explore more during training. So, if you're using a behavior policy like <span class="math-container">$\epsilon$</span>-greedy, you may want to increase your <span class="math-container">$\epsilon$</span> (i.e. probability of selecting a random action).</p></li>
  <li><p>If you estimated the state-action value function (e.g. with Q-learning), maybe you derived the policy from it by selecting the best action, but, of course, that will make your policy deterministic. You may want to use e.g. softmax to derive the policy from the state-action value function (i.e. the probability of selecting an action is proportional to its value), although Q-learning assumes that your target policy is greedy with respect to the state-action value function.</p></li>
  </ol>
  
  <p>If the optimal policy is supposed to be deterministic, then, if you find the optimal policy (which isn't probably the case), you will end up with an agent that always selects the same action. In that case, obviously, it's not a problem that the RL agent selects always the same optimal action.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-10T17:58:04.587Z
Answers.Text: Thanks ... I will have to explore how the reward can be manipulated in environments like OPen AI Gym .... For your 2nd point for policy gradients , normally e-greedy is not followed so i guess this will be more applicable for Q learning in general ..
Answers.UserDisplayName: JAYDEEP GHOSE
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-23T08:06:43.647Z
Answers.Text: the problem still remains the same - after all the modifications mentioned by you
Answers.UserDisplayName: JAYDEEP GHOSE
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-10T16:33:58.780Z
Answers.LastActivityDate: 2020-05-10T18:34:18.537Z
Answers.LastEditDate: 2020-05-10T18:34:18.537Z
Answers.UserDisplayName: nbro
Body: <p>I have seen this happening in implementations of state-of-the-art RL algorithms where the model converges to a single action over time after multiple training iterations. Are there some general loopholes or reasons why this kind of behavior is exhibited? </p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-10T15:48:50.570Z
Comments.Text: Hi and welcome to AI SE! Maybe it would be a good idea to provide an example of an algorithm that converges to one action, although I don't think a person needs this information to provide an answer (in this case).
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-10T15:51:32.163Z
Comments.Text: Hi @nbro , I have seen this happening in DQN or in policy gradient methods like PPO , so i believe there should be some generalized do's and dont's which we need to follow irrespective of algorithms to ensure that this issue doesnt happen
Comments.UserDisplayName: JAYDEEP GHOSE
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-10T15:54:33.030Z
Comments.Text: Isn't the answer to your question just "Because there's probably just an optimal action for each state"? Why do you think this is an issue? It may not be an issue. It depends. If the environment is stochastic, that may be an issue, but if the environment is deterministic, it may be the case that one action is typically the best one.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-10T15:59:15.840Z
Comments.Text: Well ..   not all environments are deterministic . For example If my agent is playing pacman and it does only one action then it might just hit into a wall and do nothing to recover . I dont think thats an optimal action and I have seen this happening as well... Correct me If I am missing anything here
Comments.UserDisplayName: JAYDEEP GHOSE
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-10T16:17:48.127Z
Comments.Text: Yes, you're right. Sometimes the policy shouldn't always select the same action. For example, if you're playing _rock paper scissors_, the optimal policy shouldn't always choose the same action, because that would make you lose against any intelligent player that recognises the pattern of the policy. However, in the cases you found, maybe there's primarily one optimal action for each state. I don't know because I don't know to which problems those algorithms that you found have been applied.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-10T16:24:56.893Z
Comments.Text: well lets take some examples .. which I faced while implementing DQN on pacman.. Now the same policy acted really well in case of a game like cart pole . But when implemented for pacman it converged to one action which lead to the agent hitting the wall all the time after 48 hours of training . Similarly , I found that if I implement the PPO algorithm for the same game , it also converged to one single action so I am assuming that there are some general guidelines which I am not following which is leading to this ..
Comments.UserDisplayName: JAYDEEP GHOSE
Tags: <reinforcement-learning><convergence><policies>
Title: Why do RL implementations converge on one action?
UserDisplayName: JAYDEEP GHOSE
ThreadText: # Why do RL implementations converge on one action?
  
  (Asked by: username_0 on 2020-05-10)
  
  I have seen this happening in implementations of state-of-the-art RL algorithms where the model converges to a single action over time after multiple training iterations. Are there some general loopholes or reasons why this kind of behavior is exhibited?
  
  **Tags:** reinforcement-learning, convergence, policies
  > **Comments:**
  > **username_1**: Hi and welcome to AI SE! Maybe it would be a good idea to provide an example of an algorithm that converges to one action, although I don't think a person needs this information to provide an answer (in this case).  
  > **username_0**: Hi @nbro , I have seen this happening in DQN or in policy gradient methods like PPO , so i believe there should be some generalized do's and dont's which we need to follow irrespective of algorithms to ensure that this issue doesnt happen  
  > **username_1**: Isn't the answer to your question just "Because there's probably just an optimal action for each state"? Why do you think this is an issue? It may not be an issue. It depends. If the environment is stochastic, that may be an issue, but if the environment is deterministic, it may be the case that one action is typically the best one.  
  > **username_0**: Well ..   not all environments are deterministic . For example If my agent is playing pacman and it does only one action then it might just hit into a wall and do nothing to recover . I dont think thats an optimal action and I have seen this happening as well... Correct me If I am missing anything here  
  > **username_1**: Yes, you're right. Sometimes the policy shouldn't always select the same action. For example, if you're playing _rock paper scissors_, the optimal policy shouldn't always choose the same action, because that would make you lose against any intelligent player that recognises the pattern of the policy. However, in the cases you found, maybe there's primarily one optimal action for each state. I don't know because I don't know to which problems those algorithms that you found have been applied.  
  > **username_0**: well lets take some examples .. which I faced while implementing DQN on pacman.. Now the same policy acted really well in case of a game like cart pole . But when implemented for pacman it converged to one action which lead to the agent hitting the wall all the time after 48 hours of training . Similarly , I found that if I implement the PPO algorithm for the same game , it also converged to one single action so I am assuming that there are some general guidelines which I am not following which is leading to this ..  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-05-10. Score: 2
  
  > Why do RL implementations converge on one action?
  
  If the optimal policy shouldn't always select the same action in the same state, i.e., if the optimal policy isn't [deterministic](https://ai.stackexchange.com/a/12275/2444) (e.g., in the case of the *rock paper scissors*, the optimal policy cannot be deterministic because any intelligent player would easily memorize your deterministic policy, so, after a while, you would always lose again that player), then there are a few things that you can do to make your policy more stochastic
  
  1. Change the reward function. If your agent ends up selecting always the same action and you don't want that, it's probably because you're not giving it the right reinforcement signal (given that the agent selects the action that apparently will give it the highest reward in the long run).
  2. Try to explore more during training. So, if you're using a behavior policy like $\epsilon$-greedy, you may want to increase your $\epsilon$ (i.e. probability of selecting a random action).
  3. If you estimated the state-action value function (e.g. with Q-learning), maybe you derived the policy from it by selecting the best action, but, of course, that will make your policy deterministic. You may want to use e.g. softmax to derive the policy from the state-action value function (i.e. the probability of selecting an action is proportional to its value), although Q-learning assumes that your target policy is greedy with respect to the state-action value function.
  
  If the optimal policy is supposed to be deterministic, then, if you find the optimal policy (which isn't probably the case), you will end up with an agent that always selects the same action. In that case, obviously, it's not a problem that the RL agent selects always the same optimal action.
  
  > **Comments:**
  > **username_0**: Thanks ... I will have to explore how the reward can be manipulated in environments like OPen AI Gym .... For your 2nd point for policy gradients , normally e-greedy is not followed so i guess this will be more applicable for Q learning in general ..  
  > **username_0**: the problem still remains the same - after all the modifications mentioned by you  
  
  ---
  
  