Answers.Body: <p>Your are correct that epsilon in epsilon-greedy and temperature parameter in the "softmax distribution" are different parameters, although they serve a similar purpose. The original author of the code has taken a small liberty with variable names in the <code>select_action</code> method in order to use just one simple name as a positional argument.</p>
  
  <blockquote>
    <p>Should the temperature be a fixed value over time? </p>
  </blockquote>
  
  <p>Not necessarily, if your goal is to converge on an optimal policy you will want to decrease temperature. A slow decay factor applied after each update or episode, as you might use for epsilon (e.g. 0.999 or other value close to 1), can also work for temperature decay. A very high temperature is roughly equivalent to epsilon of 1. As temperature becomes lower, differences in action value estimates become major differences in action selection probabilities, with a sometimes desirable effect of picking "promising" action choices more often, plus with very low probabilities of picking the actions with the worst estimates. </p>
  
  <p>Using a more sophisticated action selection such as the temperature based on in the example code can speed learning in RL. However, this particular approach is only good in some cases - it is a bit fiddly to tune, and can simply not work at all. </p>
  
  <p>The tricky part of using a temperature parameter is choosing good starting point, as well as the decay rate and end values (you have to do the last for epsilon-decay as well). The problem is that the impact of using this distribution depends on the actual differences between action choices. You need a temperature value on roughly the same scale as the Q value differences. This is difficult to figure out in advance. In addition, if the Q value differences are more pronounced in some states than in others, you risk either having next to no exploration or having too much in some parts of the problem.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-24T19:39:13.623Z
Answers.LastActivityDate: 2020-01-24T19:39:13.623Z
Answers.UserDisplayName: Neil Slater
Body: <p>This is a piece of code from my homework. </p>
  
  <pre><code># action policy: implements epsilon greedy and softmax
  def select_action(self, state, epsilon):
      qval = self.qtable[state]
      prob = []
      if (self.softmax):
          # use Softmax distribution
          prob = sp.softmax(qval / epsilon)
          #print(prob)
      else:
          # assign equal value to all actions
          prob = np.ones(self.actions) * epsilon / (self.actions -1)
          # the best action is taken with probability 1 - epsilon
          prob[np.argmax(qval)] = 1 - epsilon
      return np.random.choice(range(0, self.actions), p = prob)
  </code></pre>
  
  <p>This is a method in order to select the best action according to the two polices i think. My question is, why in the softmax computation there is the epsilon parameter used as temperature. Is really the same thing? Are they different? I think they should be two different variables. Should the temperature be a fixed value over time? Because when i use the epsilon-greedy policy my epsilon decrease over time.</p>
  
Tags: <reinforcement-learning>
Title: Is the temperature equal to epsilon in Reinforcement Learning?
UserDisplayName: FraMan
ThreadText: # Is the temperature equal to epsilon in Reinforcement Learning?
  
  (Asked by: username_0 on 2020-01-24)
  
  This is a piece of code from my homework.
  
  ```
  # action policy: implements epsilon greedy and softmax
  def select_action(self, state, epsilon):
      qval = self.qtable[state]
      prob = []
      if (self.softmax):
          # use Softmax distribution
          prob = sp.softmax(qval / epsilon)
          #print(prob)
      else:
          # assign equal value to all actions
          prob = np.ones(self.actions) * epsilon / (self.actions -1)
          # the best action is taken with probability 1 - epsilon
          prob[np.argmax(qval)] = 1 - epsilon
      return np.random.choice(range(0, self.actions), p = prob)
  
  ```
  
  This is a method in order to select the best action according to the two polices i think. My question is, why in the softmax computation there is the epsilon parameter used as temperature. Is really the same thing? Are they different? I think they should be two different variables. Should the temperature be a fixed value over time? Because when i use the epsilon-greedy policy my epsilon decrease over time.
  
  **Tags:** reinforcement-learning
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-01-24. Score: 1 | Accepted answer
  
  Your are correct that epsilon in epsilon-greedy and temperature parameter in the "softmax distribution" are different parameters, although they serve a similar purpose. The original author of the code has taken a small liberty with variable names in the `select_action` method in order to use just one simple name as a positional argument.
  
  > Should the temperature be a fixed value over time?
  
  Not necessarily, if your goal is to converge on an optimal policy you will want to decrease temperature. A slow decay factor applied after each update or episode, as you might use for epsilon (e.g. 0.999 or other value close to 1), can also work for temperature decay. A very high temperature is roughly equivalent to epsilon of 1. As temperature becomes lower, differences in action value estimates become major differences in action selection probabilities, with a sometimes desirable effect of picking "promising" action choices more often, plus with very low probabilities of picking the actions with the worst estimates.
  
  Using a more sophisticated action selection such as the temperature based on in the example code can speed learning in RL. However, this particular approach is only good in some cases - it is a bit fiddly to tune, and can simply not work at all.
  
  The tricky part of using a temperature parameter is choosing good starting point, as well as the decay rate and end values (you have to do the last for epsilon-decay as well). The problem is that the impact of using this distribution depends on the actual differences between action choices. You need a temperature value on roughly the same scale as the Q value differences. This is difficult to figure out in advance. In addition, if the Q value differences are more pronounced in some states than in others, you risk either having next to no exploration or having too much in some parts of the problem.
  
  
  ---
  
  