Answers.Body: <blockquote>
    <p>I don't fully understand what the implications of doing this will be. </p>
  </blockquote>
  
  <p>Without other matching adjustments, you will break your agent.</p>
  
  <p>The problem is how your new action space gets converted <em>back</em> into gradients to update the agent, after it has acted and needs to learn from the results. The NN component of policy function you are considering is designed to work by balancing a discrete probablility distribution. It learns by increasing the probability of actions (in the binary case, the probability of going left vs going right) that score better than a current baseline level.</p>
  
  <p>When interpreting the result from going 63.8% left, you have to resolve two things - which action did the agent take, and what changes to your parameters will increase the probability of taking that action. Unfortunately neither of these tasks are simple if you combine the action choices in the way you suggest. </p>
  
  <p>Also, you have lost exploration. The combined left/right algorithm will always output a fixed steering amount for each state. Whilst there are algorithms, like DDPG, that can work with this, it is not really possible to adapt PPO to do so.</p>
  
  <p>However, PPO already supports continuous action spaces directly. You can have your network output the mean and standard deviation of a distribution for how to steer, and sample from that. Then the action choice taken will directly relate to the output of the network and you can adjust the policy to make that choice more or less possible depending on results from taking it. If you are using a library implementation of PPO, then this option should be available to you.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-20T18:04:54.093Z
Answers.LastActivityDate: 2019-11-20T18:04:54.093Z
Answers.UserDisplayName: Neil Slater
Answers.Body: <p>Have you considered using a continuous action space? It might be worth looking into. If you aren't familiar with it, here are a few resources for discrete vs continuous action spaces - </p>
  
  <p><a href="https://mykel.kochenderfer.com/files/Kochenderfer2005ijcai2.pdf" rel="nofollow noreferrer">Modeling and Planning in Large State and Action Spaces</a></p>
  
  <p><a href="http://proceedings.mlr.press/v80/lee18b/lee18b.pdf" rel="nofollow noreferrer">Deep Reinforcement Learning in Continuous Action Spaces:a Case Study in the Game of Simulated Curling</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-20T16:00:11.820Z
Answers.Text: Can you elaborate your answer so as to avoid rather than being opinionated.
Answers.UserDisplayName: quintumnia
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-20T15:38:40.403Z
Answers.LastActivityDate: 2019-11-20T15:38:40.403Z
Answers.UserDisplayName: shuiske
Body: <p>I am training an RL agent (specifically using the <a href="https://arxiv.org/pdf/1707.06347.pdf" rel="nofollow noreferrer">PPO algorithm</a>) on a game environment with 2 possible actions <strong>left</strong> or <strong>right</strong>.</p>
  
  <p>The actions can be taken with varying "force"; e.g. go <strong>left</strong> 17% or go <strong>right</strong> 69.3%. Currently, I have the agent output 21 actions - 10 for <strong>left</strong> (in 10% increments), 10 for <strong>right</strong> in 10% increments and 1 for stay in place (do nothing). In other words, there is a direct 1-1 mapping in 10% increments between the agent output and the force the agent uses to move in the environment.</p>
  
  <p>I am wondering, if instead of outputting 21 possible actions, I change the action space to a binary output and obtain the action probabilities. The probabilities will have the form, say, [70, 30]. That is, go left with 70% probability and go right with 30% probability. Then I take these probabilities and put them through a non-linearity that translates to the actual action force taken; e.g an output of 70% probability to go left, may in fact translate to moving left with 63.8% force.</p>
  
  <p>The non linear translation is not directly observed by the agent but will determine the proceeding state, which is directly observed.</p>
  
  <p>I don't fully understand what the implications of doing this will be. Is there any argument that this would increase performance (rewards) as the agent does not need to learn direct action mappings, rather just a binary probability output?</p>
  
Tags: <reinforcement-learning>
Title: Effects of translating RL action probability through non linearity
UserDisplayName: PyRsquared
ThreadText: # Effects of translating RL action probability through non linearity
  
  (Asked by: username_0 on 2019-11-20)
  
  I am training an RL agent (specifically using the [PPO algorithm](https://arxiv.org/pdf/1707.06347.pdf)) on a game environment with 2 possible actions **left** or **right**.
  
  The actions can be taken with varying "force"; e.g. go **left** 17% or go **right** 69.3%. Currently, I have the agent output 21 actions - 10 for **left** (in 10% increments), 10 for **right** in 10% increments and 1 for stay in place (do nothing). In other words, there is a direct 1-1 mapping in 10% increments between the agent output and the force the agent uses to move in the environment.
  
  I am wondering, if instead of outputting 21 possible actions, I change the action space to a binary output and obtain the action probabilities. The probabilities will have the form, say, [70, 30]. That is, go left with 70% probability and go right with 30% probability. Then I take these probabilities and put them through a non-linearity that translates to the actual action force taken; e.g an output of 70% probability to go left, may in fact translate to moving left with 63.8% force.
  
  The non linear translation is not directly observed by the agent but will determine the proceeding state, which is directly observed.
  
  I don't fully understand what the implications of doing this will be. Is there any argument that this would increase performance (rewards) as the agent does not need to learn direct action mappings, rather just a binary probability output?
  
  **Tags:** reinforcement-learning
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2019-11-20. Score: 1 | Accepted answer
  
  > I don't fully understand what the implications of doing this will be.
  
  Without other matching adjustments, you will break your agent.
  
  The problem is how your new action space gets converted *back* into gradients to update the agent, after it has acted and needs to learn from the results. The NN component of policy function you are considering is designed to work by balancing a discrete probablility distribution. It learns by increasing the probability of actions (in the binary case, the probability of going left vs going right) that score better than a current baseline level.
  
  When interpreting the result from going 63.8% left, you have to resolve two things - which action did the agent take, and what changes to your parameters will increase the probability of taking that action. Unfortunately neither of these tasks are simple if you combine the action choices in the way you suggest.
  
  Also, you have lost exploration. The combined left/right algorithm will always output a fixed steering amount for each state. Whilst there are algorithms, like DDPG, that can work with this, it is not really possible to adapt PPO to do so.
  
  However, PPO already supports continuous action spaces directly. You can have your network output the mean and standard deviation of a distribution for how to steer, and sample from that. Then the action choice taken will directly relate to the output of the network and you can adjust the policy to make that choice more or less possible depending on results from taking it. If you are using a library implementation of PPO, then this option should be available to you.
  
  
  ---
  
  ### Answer by username_2 on 2019-11-20. Score: 1
  
  Have you considered using a continuous action space? It might be worth looking into. If you aren't familiar with it, here are a few resources for discrete vs continuous action spaces -
  
  [Modeling and Planning in Large State and Action Spaces](https://mykel.kochenderfer.com/files/Kochenderfer2005ijcai2.pdf)
  
  [Deep Reinforcement Learning in Continuous Action Spaces:a Case Study in the Game of Simulated Curling](http://proceedings.mlr.press/v80/lee18b/lee18b.pdf)
  
  > **Comments:**
  > **username_3**: Can you elaborate your answer so as to avoid rather than being opinionated.  
  
  ---
  
  