Answers.Body: <blockquote>
    <p>Where can I find sources showing that policy gradients initialize with random policies, whereas Q-Learning uses epsilon-greedy policies?</p>
  </blockquote>
  
  <p>You can find example algorithms for Q learning and policy gradients in <a href="http://incompleteideas.net/book/the-book-2nd.html" rel="nofollow noreferrer"><em>Sutton &amp; Barto's</em> Reinforcement Learning: An Introduction</a> - Q learning is in chapter 6, and policy gradients explained in chapter 13.</p>
  
  <p>Neither of these things are strictly true in all cases. However, both are very common situations for the two kinds of learning agent:</p>
  
  <ul>
  <li><p>Policy gradient solvers learn a policy function <span class="math-container">$\pi(a|s)$</span> that gives the probability of taking action <span class="math-container">$a$</span> given observed state <span class="math-container">$s$</span>. Typically this is implemented as a neural network. Neural networks are initialised randomly. For discrete action selection using softmax output layer, the initial function will roughly be choosing evenly from all possible actions. So if some actions oppose and <em>undo</em> each other, e.g. move left and move right are options - then the situation as described in your quote can easily happen. Nowadays there are many other solutions using policy gradients that don't suffer as much with this effect. For instance, deterministic policy gradient methods such as A2C or DDPG.</p></li>
  <li><p>For Q learning, and many variants of it, <span class="math-container">$\epsilon$</span>-greedy is a very commonly used action selection mechanism. It is convenient and simple, and allows a simple parameter to control balance between exploration and exploitation whilst learning. However, Q learning can work with <em>any</em> action selection mechanism that has some possibility of acting optimally at least occasionally. </p></li>
  </ul>
  
  <p>The best approach to action selection during the learning process in both policy-based methods and value-based methods is an active area of research. So if you read the RL literature you may find a lot of variation. The blog you quoted from has identified two quite common choices.</p>
  
  <blockquote>
    <p>Also, what does "max operation" have to do with epsilon-greedy policies?</p>
  </blockquote>
  
  <p>The "max operation" is finding the maximum value of some function produced when varying a parameter. The related "argmax operation" is finding the value of the parameter that produces the maximum value. Q learning can use both types of operation, but specifically uses <span class="math-container">$\text{argmax}$</span> for <span class="math-container">$\epsilon$</span>-greedy action selection.</p>
  
  <p>An <span class="math-container">$\epsilon$</span>-greedy policy requires acting optimally (according to current estimates of value function) with probability <span class="math-container">$p=1-\epsilon$</span>, and randomly with equal probability of each action with probability <span class="math-container">$p=\epsilon$</span></p>
  
  <p>In order to do this, the algorithm usually (with probability <span class="math-container">$p=1-\epsilon$</span>) needs to know what the current best guess at an optimal solution is. That is the greedy action with respect to the current Q values i.e. <span class="math-container">$\text{argmax}_a Q(s,a)$</span>  - </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-20T20:34:50.553Z
Answers.LastActivityDate: 2018-10-20T20:56:10.753Z
Answers.LastEditDate: 2018-10-20T20:56:10.753Z
Answers.UserDisplayName: Neil Slater
Body: <p>I am reviewing a statement on the website for ES regarding structured exploration. </p>
  
  <p><a href="https://blog.openai.com/evolution-strategies/" rel="nofollow noreferrer">https://blog.openai.com/evolution-strategies/</a></p>
  
  <blockquote>
    <p>Structured exploration. Some RL algorithms (especially policy
    gradients) initialize with random policies, which often manifests as
    random jitter on spot for a long time. This effect is mitigated in
    Q-Learning due to epsilon-greedy policies, where the max operation can
    cause the agents to perform some consistent action for a while (e.g.
    holding down a left arrow). This is more likely to do something in a
    game than if the agent jitters on spot, as is the case with policy
    gradients. Similar to Q-learning, ES does not suffer from these
    problems because we can use deterministic policies and achieve
    consistent exploration.</p>
  </blockquote>
  
  <p>Where can I find sources showing that policy gradients initialize with random policies, whereas Q-Learning uses epsilon-greedy policies? </p>
  
  <p>Also, what does "max operation" have to do with epsilon-greedy policies?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-10-20T19:53:39.950Z
Comments.Text: @NeilSlater I at least need an explanation that is relevant to the question. How do policy gradients initialize with random policies, whereas Q-Learning uses epsilon-greedy policies? Also, what does "max operation" have to do with epsilon-greedy policies? Thank you
Comments.UserDisplayName: FullStack
Tags: <reinforcement-learning><evolutionary-algorithms><q-learning>
Title: Some RL algorithms (especially policy gradients) initialize with random policies, which often manifests as random jitter on spot for a long time?
UserDisplayName: FullStack
ThreadText: # Some RL algorithms (especially policy gradients) initialize with random policies, which often manifests as random jitter on spot for a long time?
  
  (Asked by: username_0 on 2018-10-19)
  
  I am reviewing a statement on the website for ES regarding structured exploration.
  
  <https://blog.openai.com/evolution-strategies/>
  
  > Structured exploration. Some RL algorithms (especially policy
  > gradients) initialize with random policies, which often manifests as
  > random jitter on spot for a long time. This effect is mitigated in
  > Q-Learning due to epsilon-greedy policies, where the max operation can
  > cause the agents to perform some consistent action for a while (e.g.
  > holding down a left arrow). This is more likely to do something in a
  > game than if the agent jitters on spot, as is the case with policy
  > gradients. Similar to Q-learning, ES does not suffer from these
  > problems because we can use deterministic policies and achieve
  > consistent exploration.
  
  Where can I find sources showing that policy gradients initialize with random policies, whereas Q-Learning uses epsilon-greedy policies?
  
  Also, what does "max operation" have to do with epsilon-greedy policies?
  
  **Tags:** reinforcement-learning, evolutionary-algorithms, q-learning
  > **Comments:**
  > **username_0**: @NeilSlater I at least need an explanation that is relevant to the question. How do policy gradients initialize with random policies, whereas Q-Learning uses epsilon-greedy policies? Also, what does "max operation" have to do with epsilon-greedy policies? Thank you  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-10-20. Score: 0 | Accepted answer
  
  > Where can I find sources showing that policy gradients initialize with random policies, whereas Q-Learning uses epsilon-greedy policies?
  
  You can find example algorithms for Q learning and policy gradients in [*Sutton & Barto's* Reinforcement Learning: An Introduction](http://incompleteideas.net/book/the-book-2nd.html) - Q learning is in chapter 6, and policy gradients explained in chapter 13.
  
  Neither of these things are strictly true in all cases. However, both are very common situations for the two kinds of learning agent:
  
  * Policy gradient solvers learn a policy function $\pi(a|s)$ that gives the probability of taking action $a$ given observed state $s$. Typically this is implemented as a neural network. Neural networks are initialised randomly. For discrete action selection using softmax output layer, the initial function will roughly be choosing evenly from all possible actions. So if some actions oppose and *undo* each other, e.g. move left and move right are options - then the situation as described in your quote can easily happen. Nowadays there are many other solutions using policy gradients that don't suffer as much with this effect. For instance, deterministic policy gradient methods such as A2C or DDPG.
  * For Q learning, and many variants of it, $\epsilon$-greedy is a very commonly used action selection mechanism. It is convenient and simple, and allows a simple parameter to control balance between exploration and exploitation whilst learning. However, Q learning can work with *any* action selection mechanism that has some possibility of acting optimally at least occasionally.
  
  The best approach to action selection during the learning process in both policy-based methods and value-based methods is an active area of research. So if you read the RL literature you may find a lot of variation. The blog you quoted from has identified two quite common choices.
  
  > Also, what does "max operation" have to do with epsilon-greedy policies?
  
  The "max operation" is finding the maximum value of some function produced when varying a parameter. The related "argmax operation" is finding the value of the parameter that produces the maximum value. Q learning can use both types of operation, but specifically uses $\text{argmax}$ for $\epsilon$-greedy action selection.
  
  An $\epsilon$-greedy policy requires acting optimally (according to current estimates of value function) with probability $p=1-\epsilon$, and randomly with equal probability of each action with probability $p=\epsilon$
  
  In order to do this, the algorithm usually (with probability $p=1-\epsilon$) needs to know what the current best guess at an optimal solution is. That is the greedy action with respect to the current Q values i.e. $\text{argmax}\_a Q(s,a)$ -
  
  
  ---
  
  