Answers.Body: <p>The <a href="http://incompleteideas.net/book/first/ebook/node17.html" rel="noreferrer"><strong><span class="math-container">$\epsilon$</span>-greedy</strong></a> policy is a policy that chooses the best action (i.e. the action associated with the highest value) with probability <span class="math-container">$1-\epsilon \in [0, 1]$</span> and a random action with probability <span class="math-container">$\epsilon $</span>. The problem with <span class="math-container">$\epsilon$</span>-greedy is that, when it chooses the random actions (i.e. with probability <span class="math-container">$\epsilon$</span>), it chooses them <a href="https://en.wikipedia.org/wiki/Discrete_uniform_distribution" rel="noreferrer">uniformly</a> (i.e. it considers all actions equally good), even though certain actions (even excluding the currently best one) are better than others. Of course, this approach is not ideal in the case certain actions are extremely worse than others. Therefore, a natural solution to this problem is to select the random actions with probabilities proportional to their current values. These policies are called <a href="http://incompleteideas.net/book/first/ebook/node17.html" rel="noreferrer"><strong>softmax</strong> policies</a>.</p>
  
  <p><a href="https://en.wikipedia.org/wiki/Q-learning" rel="noreferrer"><strong>Q-learning</strong></a> is an off-policy algorithm, which means that, while learning a so-called <em>target policy</em>, it uses a so-called <em>behaviour policy</em> to select actions. The behaviour policy can either be an <span class="math-container">$\epsilon$</span>-greedy, a softmax policy or any other policy that can sufficiently explore the environment while learning. </p>
  
  <p>The figure below shows the pseudocode of the Q-learning algorithm. In this case, the <span class="math-container">$\epsilon$</span>-greedy policy is actually derived from the current estimate of the <span class="math-container">$Q$</span> function. The target policy, in this context, is represented by the <span class="math-container">$\operatorname{max}$</span> operator, which is used to select the highest <span class="math-container">$Q$</span> value of the future state <span class="math-container">$s'$</span>, which is the state the RL agent ends up in after having taken the action <span class="math-container">$a$</span> selected by the <span class="math-container">$\epsilon$</span>-greedy behaviour policy, with respect to another action <span class="math-container">$a'$</span> from state <span class="math-container">$s'$</span>. This may sound complicated, but if you read the pseudocode several times, you will understand that there are two different actions (and states). The target policy (i.e. the policy that the RL agent wants to learn) is represented by the <span class="math-container">$\operatorname{max}$</span> operator in the sense that the so-called <em>target</em> of the Q-learning update step, i.e. <span class="math-container">$r + \gamma \operatorname{max}_{a'} Q(s', a')$</span>, assumes that the <strong>greedy</strong> action is taken from the next state <span class="math-container">$s'$</span>. For this reason, Q-learning is said to learn the greedy policy (as a target policy), while using an exploratory policy, usually, the <span class="math-container">$\epsilon$</span>-greedy, but it can also be the softmax. Note that, in both cases, the policies are derived from the current estimate of the Q function. </p>
  
  <p><a href="https://i.sstatic.net/IHr3A.png" rel="noreferrer"><img src="https://i.sstatic.net/IHr3A.png" alt="enter image description here"></a></p>
  
  <p>On the other hand, <a href="https://en.wikipedia.org/wiki/State%E2%80%93action%E2%80%93reward%E2%80%93state%E2%80%93action" rel="noreferrer"><strong>SARSA</strong></a> is often considered an on-policy algorithm, given that there aren't necessarily two distinct policies, i.e. the target policy is not necessarily different than the behaviour policy, like in Q-learning (where the target policy is the greedy policy and the behaviour policy is e.g. the softmax policy derived from the current estimate of the Q function). This can more easily be seen from the pseudocode.</p>
  
  <p><a href="https://i.sstatic.net/DPT4V.png" rel="noreferrer"><img src="https://i.sstatic.net/DPT4V.png" alt="enter image description here"></a>.</p>
  
  <p>In this case, no <span class="math-container">$\operatorname{max}$</span> operator is used and the <span class="math-container">$\epsilon$</span>-greedy policy is mentioned twice: in the first case, it is used to choose the action <span class="math-container">$a$</span> and indirectly <span class="math-container">$s'$</span>, and, in the second case, to select the action <span class="math-container">$a'$</span> from <span class="math-container">$s'$</span>. In Q-learning, <span class="math-container">$a'$</span> is the action that corresponds to the highest Q value from <span class="math-container">$s'$</span> (i.e. the greedy action). Clearly, you are free to choose a different policy than the <span class="math-container">$\epsilon$</span>-greedy (in both cases), but this will possibly have a different effect.</p>
  
  <p>To conclude, to understand the difference between Q-learning and SARSA and the places where the <span class="math-container">$\epsilon$</span>-greedy or softmax policies can be used, it is better to look at the pseudocode.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-26T12:38:10.447Z
Answers.Text: If I train my agent with softmax policy, is the learning process more unstable w.r.t the epsilon greedy?
Answers.UserDisplayName: FraMan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-26T12:45:00.337Z
Answers.Text: @FraMan [http://incompleteideas.net/book/first/ebook/node17.html](http://incompleteideas.net/book/first/ebook/node17.html) says that it is unclear whether softmax is better or not than $\epsilon$-greedy and it may depend on the problem.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-26T15:30:04.873Z
Answers.Text: does Q-Learning + epsilon greedy makes sense? Q-Learning use a greedy policy as default.
Answers.UserDisplayName: FraMan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-26T15:32:12.573Z
Answers.Text: @FraMan I think this is a good new question you could ask on the website ;)
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-22T22:24:11.983Z
Answers.LastActivityDate: 2020-01-22T22:29:13.530Z
Answers.LastEditDate: 2020-01-22T22:29:13.530Z
Answers.UserDisplayName: nbro
Body: <p>Could someone explain to me which is the key difference between the <span class="math-container">$\epsilon$</span>-greedy policy and the softmax policy? In particular, in the contest of SARSA and Q-Learning algorithms. I understood the main difference between these two algorithms, but I didn't understand all the combinations between algorithm and policy</p>
  <ul>
  <li>SARSA + <span class="math-container">$\epsilon$</span>-greedy</li>
  <li>SARSA + Softmax</li>
  <li>Q-Learning + <span class="math-container">$\epsilon$</span>-greedy</li>
  <li>Q-Learning + Softmax</li>
  </ul>
  
Tags: <reinforcement-learning><q-learning><sarsa><epsilon-greedy-policy><softmax-policy>
Title: What is the difference between the $\epsilon$-greedy and softmax policies?
UserDisplayName: FraMan
ThreadText: # What is the difference between the $\epsilon$-greedy and softmax policies?
  
  (Asked by: username_0 on 2020-01-21)
  
  Could someone explain to me which is the key difference between the $\epsilon$-greedy policy and the softmax policy? In particular, in the contest of SARSA and Q-Learning algorithms. I understood the main difference between these two algorithms, but I didn't understand all the combinations between algorithm and policy
  
  * SARSA + $\epsilon$-greedy
  * SARSA + Softmax
  * Q-Learning + $\epsilon$-greedy
  * Q-Learning + Softmax
  
  **Tags:** reinforcement-learning, q-learning, sarsa, epsilon-greedy-policy, softmax-policy
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-01-22. Score: 7
  
  The [**$\epsilon$-greedy**](http://incompleteideas.net/book/first/ebook/node17.html) policy is a policy that chooses the best action (i.e. the action associated with the highest value) with probability $1-\epsilon \in [0, 1]$ and a random action with probability $\epsilon $. The problem with $\epsilon$-greedy is that, when it chooses the random actions (i.e. with probability $\epsilon$), it chooses them [uniformly](https://en.wikipedia.org/wiki/Discrete_uniform_distribution) (i.e. it considers all actions equally good), even though certain actions (even excluding the currently best one) are better than others. Of course, this approach is not ideal in the case certain actions are extremely worse than others. Therefore, a natural solution to this problem is to select the random actions with probabilities proportional to their current values. These policies are called [**softmax** policies](http://incompleteideas.net/book/first/ebook/node17.html).
  
  [**Q-learning**](https://en.wikipedia.org/wiki/Q-learning) is an off-policy algorithm, which means that, while learning a so-called *target policy*, it uses a so-called *behaviour policy* to select actions. The behaviour policy can either be an $\epsilon$-greedy, a softmax policy or any other policy that can sufficiently explore the environment while learning.
  
  The figure below shows the pseudocode of the Q-learning algorithm. In this case, the $\epsilon$-greedy policy is actually derived from the current estimate of the $Q$ function. The target policy, in this context, is represented by the $\operatorname{max}$ operator, which is used to select the highest $Q$ value of the future state $s'$, which is the state the RL agent ends up in after having taken the action $a$ selected by the $\epsilon$-greedy behaviour policy, with respect to another action $a'$ from state $s'$. This may sound complicated, but if you read the pseudocode several times, you will understand that there are two different actions (and states). The target policy (i.e. the policy that the RL agent wants to learn) is represented by the $\operatorname{max}$ operator in the sense that the so-called *target* of the Q-learning update step, i.e. $r + \gamma \operatorname{max}\_{a'} Q(s', a')$, assumes that the **greedy** action is taken from the next state $s'$. For this reason, Q-learning is said to learn the greedy policy (as a target policy), while using an exploratory policy, usually, the $\epsilon$-greedy, but it can also be the softmax. Note that, in both cases, the policies are derived from the current estimate of the Q function.
  
  [![enter image description here](https://i.sstatic.net/IHr3A.png)](https://i.sstatic.net/IHr3A.png)
  
  On the other hand, [**SARSA**](https://en.wikipedia.org/wiki/State%E2%80%93action%E2%80%93reward%E2%80%93state%E2%80%93action) is often considered an on-policy algorithm, given that there aren't necessarily two distinct policies, i.e. the target policy is not necessarily different than the behaviour policy, like in Q-learning (where the target policy is the greedy policy and the behaviour policy is e.g. the softmax policy derived from the current estimate of the Q function). This can more easily be seen from the pseudocode.
  
  [![enter image description here](https://i.sstatic.net/DPT4V.png)](https://i.sstatic.net/DPT4V.png).
  
  In this case, no $\operatorname{max}$ operator is used and the $\epsilon$-greedy policy is mentioned twice: in the first case, it is used to choose the action $a$ and indirectly $s'$, and, in the second case, to select the action $a'$ from $s'$. In Q-learning, $a'$ is the action that corresponds to the highest Q value from $s'$ (i.e. the greedy action). Clearly, you are free to choose a different policy than the $\epsilon$-greedy (in both cases), but this will possibly have a different effect.
  
  To conclude, to understand the difference between Q-learning and SARSA and the places where the $\epsilon$-greedy or softmax policies can be used, it is better to look at the pseudocode.
  
  > **Comments:**
  > **username_0**: If I train my agent with softmax policy, is the learning process more unstable w.r.t the epsilon greedy?  
  > **username_1**: @username_0 [http://incompleteideas.net/book/first/ebook/node17.html](http://incompleteideas.net/book/first/ebook/node17.html) says that it is unclear whether softmax is better or not than $\epsilon$-greedy and it may depend on the problem.  
  > **username_0**: does Q-Learning + epsilon greedy makes sense? Q-Learning use a greedy policy as default.  
  > **username_1**: @username_0 I think this is a good new question you could ask on the website ;)  
  
  ---
  
  