Answers.Body: <blockquote>
    <p>I believe that if I follow the policy (sample an action from the policy) I make use of exploration because each action has a certain probability so I will explore all actions for a given state.</p>
  </blockquote>
  
  <p>Yes, having a stochastic policy function is the main way that a lot of policy gradient methods achieve exploration, including REINFORCE, A2C, A3C.</p>
  
  <blockquote>
    <p>Is it beneficial or is it common to use extra exploration strategy like UCB, Thompson sampling etc. n such algorithms?</p>
  </blockquote>
  
  <p>It can be, but needs to be done carefully, as the gradient sampling for the policy function is different. Many policy gradient methods are strictly on-policy and will not work if you simply add extra exploration. It is relatively straightforward to adjust the critic part of actor-critic methods by using e.g.  Q learning update rules for it. However, the gradient of the policy function is trickier.</p>
  
  <p>There are some policy gradient methods that do work with a separate, tunable, exploration function. <a href="https://towardsdatascience.com/deep-deterministic-policy-gradients-explained-2d94655a9b7b" rel="nofollow noreferrer">Deep Deterministic Policy Gradient</a> (DDPG) may be of interest to you - as per the title, it works with a deterministic policy function, and exploration is achieved by adding a separate noise function on top. The sampling for policy gradient is then corrected for being off-policy.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-07T09:28:32.973Z
Answers.Text: Note this is an area where I have not delved into the maths in any depth, so I cannot help with questions about applicability or implementation of any *specific* noise function.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-07T09:27:36.903Z
Answers.LastActivityDate: 2020-06-07T09:27:36.903Z
Answers.UserDisplayName: Neil Slater
Answers.Body: <p>Neil Slater's answer is very nice, but I have a couple more suggestions: </p>
  
  <ul>
  <li><p>You can use entropy regularization. Basically, you modify your loss function to penalize low policy entropy (so less loss for more entropy) which should prevent your policy from becoming "too deterministic" too early. </p></li>
  <li><p>You can also try maximum-entropy methods, like SAC, which employ a different strategy for promoting policy entropy. </p></li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-12T15:15:15.963Z
Answers.LastActivityDate: 2020-06-12T15:15:15.963Z
Answers.UserDisplayName: harwiltz
Body: <p>In policy gradient algorithms the output is a stochastic policy - a probability for each action.</p>
  
  <p>I believe that if I follow the policy (sample an action from the policy) I make use of exploration because each action has a certain probability so I will explore all actions for a given state.</p>
  
  <p>Is it beneficial or is it common to use extra exploration strategies, like UCB, Thompson sampling, etc. with such algorithms?</p>
  
Tags: <reinforcement-learning><policy-gradients><exploration-exploitation-tradeoff><upper-confidence-bound><thompson-sampling>
Title: Should I use exploration strategy in Policy Gradient algorithms?
UserDisplayName: gnikol
ThreadText: # Should I use exploration strategy in Policy Gradient algorithms?
  
  (Asked by: username_0 on 2020-06-06)
  
  In policy gradient algorithms the output is a stochastic policy - a probability for each action.
  
  I believe that if I follow the policy (sample an action from the policy) I make use of exploration because each action has a certain probability so I will explore all actions for a given state.
  
  Is it beneficial or is it common to use extra exploration strategies, like UCB, Thompson sampling, etc. with such algorithms?
  
  **Tags:** reinforcement-learning, policy-gradients, exploration-exploitation-tradeoff, upper-confidence-bound, thompson-sampling
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2020-06-07. Score: 2 | Accepted answer
  
  > I believe that if I follow the policy (sample an action from the policy) I make use of exploration because each action has a certain probability so I will explore all actions for a given state.
  
  Yes, having a stochastic policy function is the main way that a lot of policy gradient methods achieve exploration, including REINFORCE, A2C, A3C.
  
  > Is it beneficial or is it common to use extra exploration strategy like UCB, Thompson sampling etc. n such algorithms?
  
  It can be, but needs to be done carefully, as the gradient sampling for the policy function is different. Many policy gradient methods are strictly on-policy and will not work if you simply add extra exploration. It is relatively straightforward to adjust the critic part of actor-critic methods by using e.g. Q learning update rules for it. However, the gradient of the policy function is trickier.
  
  There are some policy gradient methods that do work with a separate, tunable, exploration function. [Deep Deterministic Policy Gradient](https://towardsdatascience.com/deep-deterministic-policy-gradients-explained-2d94655a9b7b) (DDPG) may be of interest to you - as per the title, it works with a deterministic policy function, and exploration is achieved by adding a separate noise function on top. The sampling for policy gradient is then corrected for being off-policy.
  
  > **Comments:**
  > **username_1**: Note this is an area where I have not delved into the maths in any depth, so I cannot help with questions about applicability or implementation of any *specific* noise function.  
  
  ---
  
  ### Answer by username_2 on 2020-06-12. Score: 2
  
  username_1 username_1's answer is very nice, but I have a couple more suggestions:
  
  * You can use entropy regularization. Basically, you modify your loss function to penalize low policy entropy (so less loss for more entropy) which should prevent your policy from becoming "too deterministic" too early.
  * You can also try maximum-entropy methods, like SAC, which employ a different strategy for promoting policy entropy.
  
  
  ---
  
  