Body: <p>In a continuous action space (for instance, in PPO, TRPO, REINFORCE, etc.), during training, an action is sampled from the random distribution with <span class="math-container">$\mu$</span> and <span class="math-container">$\sigma$</span>. This results in an inherent exploration. However, during testing, when we no longer need to explore but exploit, the action should be deterministic, i.e. just <span class="math-container">$\mu$</span>, right?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-09T16:53:41.577Z
Comments.Text: Which specific RL algorithm are you referring to? Please, edit your post to say this explicitly and provide a link to the paper. Moreover, please, put your **specific** question in the title. "Action selection during testing in continuous action space" is not a question but a topic/problem.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-09T16:56:25.710Z
Comments.Text: Proximal Policy Optimization would be one example. So, some algorithm where the policy network generates action values instead of state-value estimates or the like.
Comments.UserDisplayName: Daniel B.
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-09T17:02:57.767Z
Comments.Text: @DanielB. Ok, let's see what the OP is referring to. I think that a little bit more of context is needed to answer this question properly.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-09T22:05:59.067Z
Comments.Text: @nbro, I meant all algorithms that sample their actions using the given $\mu$ and $\sigma$, i.e. Reinforce, A2C, A3C, TRPO, PPO, SAC unlike DDPG and TD3 which return just $\mu$ as their action. But for clarity I edited the my question and made it for PPO. And thank you for improving my question.
Comments.UserDisplayName: Mika
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-10T02:10:30.283Z
Comments.Text: Btw, a thorough explanation of the procedure for sampling actions from a distribution parameterized by a network's predictions/outputs is provided in Section 6 of this [paper](https://link.springer.com/content/pdf/10.1007/BF00992696.pdf). Might be that this paper also contains the answer to your question. Haven't read it to the end yet, though. So, I can't tell.
Comments.UserDisplayName: Daniel B.
Tags: <reinforcement-learning><proximal-policy-optimization><reinforce><continuous-action-spaces><exploration-strategies>
Title: Are actions deterministic during testing in continuous action space PPO?
UserDisplayName: Mika
ThreadText: # Are actions deterministic during testing in continuous action space PPO?
  
  (Asked by: username_0 on 2021-01-09)
  
  In a continuous action space (for instance, in PPO, TRPO, REINFORCE, etc.), during training, an action is sampled from the random distribution with $\mu$ and $\sigma$. This results in an inherent exploration. However, during testing, when we no longer need to explore but exploit, the action should be deterministic, i.e. just $\mu$, right?
  
  **Tags:** reinforcement-learning, proximal-policy-optimization, reinforce, continuous-action-spaces, exploration-strategies
  > **Comments:**
  > **username_1**: Which specific RL algorithm are you referring to? Please, edit your post to say this explicitly and provide a link to the paper. Moreover, please, put your **specific** question in the title. "Action selection during testing in continuous action space" is not a question but a topic/problem.  
  > **username_2**: Proximal Policy Optimization would be one example. So, some algorithm where the policy network generates action values instead of state-value estimates or the like.  
  > **username_1**: @DanielB. Ok, let's see what the OP is referring to. I think that a little bit more of context is needed to answer this question properly.  
  > **username_0**: @nbro, I meant all algorithms that sample their actions using the given $\mu$ and $\sigma$, i.e. Reinforce, A2C, A3C, TRPO, PPO, SAC unlike DDPG and TD3 which return just $\mu$ as their action. But for clarity I edited the my question and made it for PPO. And thank you for improving my question.  
  > **username_2**: Btw, a thorough explanation of the procedure for sampling actions from a distribution parameterized by a network's predictions/outputs is provided in Section 6 of this [paper](https://link.springer.com/content/pdf/10.1007/BF00992696.pdf). Might be that this paper also contains the answer to your question. Haven't read it to the end yet, though. So, I can't tell.  
  
  ---
  
  ## 0 answers
  
  