Answers.Body: <p>I have not personally worked enough with continuous action spaces to be capable of confidently giving advise based on my own experience, but I can point you to likely relevant research (more recent than the research you already pointed to yourself):</p>
  
  <hr>
  
  <p>The most common / "popular" area of research in recent years that involves RL and continuous action spaces uses robot / physics simulators such as MuJoCo. Some examples:</p>
  
  <ul>
  <li><a href="https://arxiv.org/abs/1602.01783" rel="nofollow noreferrer">Asynchronous Methods for Deep Reinforcement Learning</a> mentions using an entropy cost term in the loss function used for training to encourage exploration (see Supplementary Material after References).</li>
  <li><a href="https://arxiv.org/abs/1706.01905" rel="nofollow noreferrer">Parameter Space Noise for Exploration</a> describes the idea of adding noise directly to the learned parameters on the Neural Network. That way it basically always predicts a different action to be "optimal" (due to the noise), and therefore a "greedy" policy based on the noisy parameters will in fact have exploration and not be fully greedy.</li>
  <li><a href="https://arxiv.org/abs/1801.00690" rel="nofollow noreferrer">DeepMind Control Suite</a> is a somewhat recent paper that proposes a suite of benchmarks for continuous control problems. It is filled with references to relevant papers describing all kinds of algorithms, and hopefully every single one of those papers would also describe how they perform exploration.</li>
  </ul>
  
  <hr>
  
  <p>Recently, at the ICML 2018 conference, there was a complete workshop dedicated to Exploration in Reinforcement Learning. <a href="https://sites.google.com/view/erl-2018/accepted-papers" rel="nofollow noreferrer">Here is the list of accepted papers at this workshop</a>. Note that it's about Exploration in RL, not only about continuous action spaces, so there might be papers in there that are only applicable to discrete action spaces. Nevertheless, I would be highly surprised if there's nothing in there that's relevant.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-09T21:05:02.170Z
Answers.Text: Thanks for the list of places to look. I like the idea of doing exploration by perturbing the weights of the Neural Network.  This type of method can be used in discrete or continuous action spaces and has the added benefit of allowing random logic (inside the NN) to be tried for a while, then shifted back if it doesn't help.  A method similar to the one cited above can be found in [NOISY NETWORKS FOR EXPLORATION](https://arxiv.org/abs/1706.10295) by the Google Deep Mind team.
Answers.UserDisplayName: Jed
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-06T13:44:08.937Z
Answers.LastActivityDate: 2018-08-06T13:44:08.937Z
Answers.UserDisplayName: Dennis Soemers
Answers.Body: <p>Firstly, note that the Gaussian policies you describe <em>are not</em> equivalent to <span class="math-container">$\epsilon$</span>-greedy, mainly for this reason: for a fixed policy, the policy's variance in the Gaussian case depends on the state, while in the <span class="math-container">$\epsilon$</span>-greedy case it does not. Right off the bat, the Gaussian policy should achieve less regret than <span class="math-container">$\epsilon$</span>-greedy.</p>
  <p>Other approaches to exploration in continuous action spaces include :</p>
  <ul>
  <li>Parameterizing the policy differently. You're not limited to gaussians, rather any parameterizable distribution (particularly those that can be reparameterized according to the reparameterization trick) will do.</li>
  <li>Using an entropy bonus. You can subtract your policy's entropy in the expression for your loss function, which helps prevent your policy from becoming &quot;too deterministic&quot; before the agent learns the environment sufficiently.</li>
  <li>Surprise/curiosity based methods. From this I mean methods that do reward shaping based on some measure of uncertainty in the policy -- at each transition, this measure of uncertainty is added to the reward. See &quot;Exploration by Random Network Distillation&quot; for example.</li>
  <li>Maximum Entropy methods. These have slightly different objectives than standard RL that also emphasize policy entropy, so they should promote exploration. See SAC for example.</li>
  <li>Use deterministic policy gradients. Then you can literally apply <span class="math-container">$\epsilon$</span>-greedy if you want, or simply add noise to the output of the policy. See TD3 for example.</li>
  </ul>
  <p>I doubt this is an exhaustive list, but I hope it helps.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-02T14:43:38.580Z
Answers.LastActivityDate: 2020-10-02T14:43:38.580Z
Answers.UserDisplayName: harwiltz
Answers.Body: <p>These papers may give a solution to exploration in continuous action space.</p>
  <p>Qurey Q-value to exploration like espilon greedy method in discrete action space. <a href="https://arxiv.org/abs/2308.11348" rel="nofollow noreferrer">https://arxiv.org/abs/2308.11348</a>.</p>
  <p>Predict a better policy for exploration. <a href="https://arxiv.org/abs/1910.12807" rel="nofollow noreferrer">https://arxiv.org/abs/1910.12807</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-08-05T05:38:38.687Z
Answers.LastActivityDate: 2024-08-05T05:51:43.487Z
Answers.LastEditDate: 2024-08-05T05:51:43.487Z
Answers.UserDisplayName: raincchio
Body: <p>I'm building a deep neural network to serve as the policy estimator in an actor-critic reinforcement learning algorithm for a continuing (not episodic) case.  I'm trying to determine how to explore the action space. I have read through <a href="http://incompleteideas.net/book/the-book-2nd.html" rel="nofollow noreferrer">this text book</a> by Sutton, and, in section 13.7, he gives one way to explore a continuous action space.  In essence, you train the policy model to give a mean and standard deviation as an output, so you can sample a value from that Gaussian distribution to pick an action.  This just seems like the continuous action-space equivalent of an <span class="math-container">$\epsilon$</span>-greedy policy.</p>
  <p><em>Are there other continuous action space exploration strategies I should consider?</em></p>
  <p>I've been doing some research online and found some articles related to RL in robotics and found that the <a href="https://papers.nips.cc/paper/3545-policy-search-for-motor-primitives-in-robotics.pdf" rel="nofollow noreferrer">PoWER</a> and <a href="https://arxiv.org/pdf/1206.4621" rel="nofollow noreferrer">PI^2</a> algorithms do something similar to what is in the textbook.</p>
  <p><em>Are these, or other, algorithms &quot;better&quot; (obviously depends on the problem being solved) alternatives to what is listed in the textbook for continuous action-space problems?</em></p>
  <p>I know that this question could have many answers, but I'm just looking for a reasonably short list of options that people have used in real applications that work.</p>
  
Tags: <neural-networks><reinforcement-learning><reference-request><actor-critic-methods>
Title: What are the available exploration strategies for continuous action space scenarios in RL?
UserDisplayName: Jed
ThreadText: # What are the available exploration strategies for continuous action space scenarios in RL?
  
  (Asked by: username_0 on 2018-08-03)
  
  I'm building a deep neural network to serve as the policy estimator in an actor-critic reinforcement learning algorithm for a continuing (not episodic) case. I'm trying to determine how to explore the action space. I have read through [this text book](http://incompleteideas.net/book/the-book-2nd.html) by Sutton, and, in section 13.7, he gives one way to explore a continuous action space. In essence, you train the policy model to give a mean and standard deviation as an output, so you can sample a value from that Gaussian distribution to pick an action. This just seems like the continuous action-space equivalent of an $\epsilon$-greedy policy.
  
  *Are there other continuous action space exploration strategies I should consider?*
  
  I've been doing some research online and found some articles related to RL in robotics and found that the [PoWER](https://papers.nips.cc/paper/3545-policy-search-for-motor-primitives-in-robotics.pdf) and [PI^2](https://arxiv.org/pdf/1206.4621) algorithms do something similar to what is in the textbook.
  
  *Are these, or other, algorithms "better" (obviously depends on the problem being solved) alternatives to what is listed in the textbook for continuous action-space problems?*
  
  I know that this question could have many answers, but I'm just looking for a reasonably short list of options that people have used in real applications that work.
  
  **Tags:** neural-networks, reinforcement-learning, reference-request, actor-critic-methods
  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_1 on 2018-08-06. Score: 1 | Accepted answer
  
  I have not personally worked enough with continuous action spaces to be capable of confidently giving advise based on my own experience, but I can point you to likely relevant research (more recent than the research you already pointed to yourself):
  
  ---
  
  The most common / "popular" area of research in recent years that involves RL and continuous action spaces uses robot / physics simulators such as MuJoCo. Some examples:
  
  * [Asynchronous Methods for Deep Reinforcement Learning](https://arxiv.org/abs/1602.01783) mentions using an entropy cost term in the loss function used for training to encourage exploration (see Supplementary Material after References).
  * [Parameter Space Noise for Exploration](https://arxiv.org/abs/1706.01905) describes the idea of adding noise directly to the learned parameters on the Neural Network. That way it basically always predicts a different action to be "optimal" (due to the noise), and therefore a "greedy" policy based on the noisy parameters will in fact have exploration and not be fully greedy.
  * [DeepMind Control Suite](https://arxiv.org/abs/1801.00690) is a somewhat recent paper that proposes a suite of benchmarks for continuous control problems. It is filled with references to relevant papers describing all kinds of algorithms, and hopefully every single one of those papers would also describe how they perform exploration.
  
  ---
  
  Recently, at the ICML 2018 conference, there was a complete workshop dedicated to Exploration in Reinforcement Learning. [Here is the list of accepted papers at this workshop](https://sites.google.com/view/erl-2018/accepted-papers). Note that it's about Exploration in RL, not only about continuous action spaces, so there might be papers in there that are only applicable to discrete action spaces. Nevertheless, I would be highly surprised if there's nothing in there that's relevant.
  
  > **Comments:**
  > **username_0**: Thanks for the list of places to look. I like the idea of doing exploration by perturbing the weights of the Neural Network.  This type of method can be used in discrete or continuous action spaces and has the added benefit of allowing random logic (inside the NN) to be tried for a while, then shifted back if it doesn't help.  A method similar to the one cited above can be found in [NOISY NETWORKS FOR EXPLORATION](https://arxiv.org/abs/1706.10295) by the Google Deep Mind team.  
  
  ---
  
  ### Answer by username_2 on 2020-10-02. Score: 1
  
  Firstly, note that the Gaussian policies you describe *are not* equivalent to $\epsilon$-greedy, mainly for this reason: for a fixed policy, the policy's variance in the Gaussian case depends on the state, while in the $\epsilon$-greedy case it does not. Right off the bat, the Gaussian policy should achieve less regret than $\epsilon$-greedy.
  
  Other approaches to exploration in continuous action spaces include :
  
  * Parameterizing the policy differently. You're not limited to gaussians, rather any parameterizable distribution (particularly those that can be reparameterized according to the reparameterization trick) will do.
  * Using an entropy bonus. You can subtract your policy's entropy in the expression for your loss function, which helps prevent your policy from becoming "too deterministic" before the agent learns the environment sufficiently.
  * Surprise/curiosity based methods. From this I mean methods that do reward shaping based on some measure of uncertainty in the policy -- at each transition, this measure of uncertainty is added to the reward. See "Exploration by Random Network Distillation" for example.
  * Maximum Entropy methods. These have slightly different objectives than standard RL that also emphasize policy entropy, so they should promote exploration. See SAC for example.
  * Use deterministic policy gradients. Then you can literally apply $\epsilon$-greedy if you want, or simply add noise to the output of the policy. See TD3 for example.
  
  I doubt this is an exhaustive list, but I hope it helps.
  
  
  ---
  
  ### Answer by username_3 on 2024-08-05. Score: 0
  
  These papers may give a solution to exploration in continuous action space.
  
  Qurey Q-value to exploration like espilon greedy method in discrete action space. <https://arxiv.org/abs/2308.11348>.
  
  Predict a better policy for exploration. <https://arxiv.org/abs/1910.12807>.
  
  
  ---
  
  