Answers.Body: <blockquote>
  <p>DQN on the other hand, explores using epsilon greedy exploration. Either selecting the best action or a random action.</p>
  </blockquote>
  <p>This is a very common choice, because it is simple to implement and quite robust. However, it is not a requirement of DQN. You can use other action choice mechanisms, <em>provided</em> all choices are covered with a non-zero probability of being selected.</p>
  <blockquote>
  <p>What if we use a softmax function to select the next action in DQN? Does that provide better exploration and policy convergence?</p>
  </blockquote>
  <p>It might in some circumstances. A key benefit is that it will tend to focus on action choices that are close to its current best guess at optimal. One problem is that if there is a large enough error in Q value estimates, it can get stuck as the exploration could heavily favour a current best value estimate. For instance, if one estimate is accurate and relatively high, but another estimate is much lower but in reality would be a good action choice, then the softmax probabilities to resample the bad estimate will be very low and it could take a very long time to fix.</p>
  <p>A more major problem is that the Q values are not independent logits that define preferences (whilst they would be in a Policy Gradient approach). The Q values have an inherent meaning and scale based on summed rewards. Which means that differences between optimal and non-optimal Q value estimates could be at any scale, maybe just 0.1 difference in value, or maybe 100 or more. This makes plain softmax a poor choice - it might suggest a near random exploration policy in one problem, and a near determinitsic policy in another, irrespective of what exploration might be useful at the current stage of learning.</p>
  <p>A fix for this is to use Gibbs/Boltzmann action selection, which modifies softmax by adding a scaling factor - often called <em>temperature</em> and noted as <span class="math-container">$T$</span> - to adjust the relative scale between action choices:</p>
  <p><span class="math-container">$$\pi(a|s) = \frac{e^{q(s,a)/T}}{\sum_{x \in \mathcal{A}} e^{q(s,x)/T}}$$</span></p>
  <p>This can work nicely to focus later exploration towards refining differences between actions that are likely to be good whilst only rarely making obvious mistakes. However it comes at a cost - you have to decide starting <span class="math-container">$T$</span>, the rate to decay <span class="math-container">$T$</span> and an end value of <span class="math-container">$T$</span>. A rough idea of min/max action value that the agent is likely to estimate can help.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-12T19:18:21.610Z
Answers.Text: So, using softmax could be considered a substitute for other exploration trick?
Answers.UserDisplayName: Hermes Morales
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-12T19:49:25.480Z
Answers.Text: @HermesMorales: Yes it is a way to construct a flexible exploring policy
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-23T16:59:49.110Z
Answers.LastActivityDate: 2020-06-23T17:06:40.560Z
Answers.LastEditDate: 2020-06-23T17:06:40.560Z
Answers.UserDisplayName: Neil Slater
Body: <p>I understand the two major branches of RL are Q-Learning and Policy Gradient methods.</p>
  <p>From my understanding (correct me if I'm wrong), policy gradient methods have an inherent exploration built-in as it selects actions using a probability distribution.</p>
  <p>On the other hand, DQN explores using the <span class="math-container">$\epsilon$</span>-greedy policy. Either selecting the best action or a random action.</p>
  <p>What if we use a softmax function to select the next action in DQN? Does that provide better exploration and policy convergence?</p>
  
Tags: <reinforcement-learning><dqn><policy-gradients><epsilon-greedy-policy><softmax-policy>
Title: What happens when you select actions using softmax instead of epsilon greedy in DQN?
UserDisplayName: Linsu Han
ThreadText: # What happens when you select actions using softmax instead of epsilon greedy in DQN?
  
  (Asked by: username_0 on 2020-06-23)
  
  I understand the two major branches of RL are Q-Learning and Policy Gradient methods.
  
  From my understanding (correct me if I'm wrong), policy gradient methods have an inherent exploration built-in as it selects actions using a probability distribution.
  
  On the other hand, DQN explores using the $\epsilon$-greedy policy. Either selecting the best action or a random action.
  
  What if we use a softmax function to select the next action in DQN? Does that provide better exploration and policy convergence?
  
  **Tags:** reinforcement-learning, dqn, policy-gradients, epsilon-greedy-policy, softmax-policy
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-06-23. Score: 4 | Accepted answer
  
  > DQN on the other hand, explores using epsilon greedy exploration. Either selecting the best action or a random action.
  
  This is a very common choice, because it is simple to implement and quite robust. However, it is not a requirement of DQN. You can use other action choice mechanisms, *provided* all choices are covered with a non-zero probability of being selected.
  
  > What if we use a softmax function to select the next action in DQN? Does that provide better exploration and policy convergence?
  
  It might in some circumstances. A key benefit is that it will tend to focus on action choices that are close to its current best guess at optimal. One problem is that if there is a large enough error in Q value estimates, it can get stuck as the exploration could heavily favour a current best value estimate. For instance, if one estimate is accurate and relatively high, but another estimate is much lower but in reality would be a good action choice, then the softmax probabilities to resample the bad estimate will be very low and it could take a very long time to fix.
  
  A more major problem is that the Q values are not independent logits that define preferences (whilst they would be in a Policy Gradient approach). The Q values have an inherent meaning and scale based on summed rewards. Which means that differences between optimal and non-optimal Q value estimates could be at any scale, maybe just 0.1 difference in value, or maybe 100 or more. This makes plain softmax a poor choice - it might suggest a near random exploration policy in one problem, and a near determinitsic policy in another, irrespective of what exploration might be useful at the current stage of learning.
  
  A fix for this is to use Gibbs/Boltzmann action selection, which modifies softmax by adding a scaling factor - often called *temperature* and noted as $T$ - to adjust the relative scale between action choices:
  
  $$\pi(a|s) = \frac{e^{q(s,a)/T}}{\sum\_{x \in \mathcal{A}} e^{q(s,x)/T}}$$
  
  This can work nicely to focus later exploration towards refining differences between actions that are likely to be good whilst only rarely making obvious mistakes. However it comes at a cost - you have to decide starting $T$, the rate to decay $T$ and an end value of $T$. A rough idea of min/max action value that the agent is likely to estimate can help.
  
  > **Comments:**
  > **username_2**: So, using softmax could be considered a substitute for other exploration trick?  
  > **username_1**: @HermesMorales: Yes it is a way to construct a flexible exploring policy  
  
  ---
  
  