Answers.Body: <p>Do you have to use Boltzmann exploration, strictly? There is a modification for Boltzmann exploration called Mellow-max. It, basically, provides an adaptive temperature for Boltzmann exploration.</p>
  <p>Here is the link for the paper for tuning mellow-max with deep reinforcement learning (DQN is often mentioned): <a href="http://cs.brown.edu/people/gdk/pubs/tuning_mellowmax_drlw.pdf" rel="nofollow noreferrer">http://cs.brown.edu/people/gdk/pubs/tuning_mellowmax_drlw.pdf</a></p>
  <p>Here is the link for mellow-max implemented with SARSA (I recommend reading this first, to get an understanding of mellow-max): <a href="https://arxiv.org/pdf/1612.05628.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1612.05628.pdf</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-26T18:55:18.503Z
Answers.LastActivityDate: 2020-06-26T18:55:18.503Z
Answers.UserDisplayName: rert588
Body: <p>I'm currently modeling DQN in Reinforcement Learning. My question is: what are the best practices related to Boltzmann Exploration? My current thoughts are: (1) Let the temperature decay through training and finally stop at 0.01, when the method will always select the best practice, with almost no randomness. (2) Standardize the predicted Q values before feeding into the softmax function.</p>
  <p>Currently, I'm using (2), and the reward is suffering from high variance. I'm wondering whether it has something to do with the exploration method?</p>
  
Tags: <neural-networks><machine-learning><reinforcement-learning><deep-learning><q-learning>
Title: What's the best practice for Boltzmann Exploration temperature in RL?
UserDisplayName: o_yeah
ThreadText: # What's the best practice for Boltzmann Exploration temperature in RL?
  
  (Asked by: username_0 on 2020-06-21)
  
  I'm currently modeling DQN in Reinforcement Learning. My question is: what are the best practices related to Boltzmann Exploration? My current thoughts are: (1) Let the temperature decay through training and finally stop at 0.01, when the method will always select the best practice, with almost no randomness. (2) Standardize the predicted Q values before feeding into the softmax function.
  
  Currently, I'm using (2), and the reward is suffering from high variance. I'm wondering whether it has something to do with the exploration method?
  
  **Tags:** neural-networks, machine-learning, reinforcement-learning, deep-learning, q-learning
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-06-26. Score: 1 | Accepted answer
  
  Do you have to use Boltzmann exploration, strictly? There is a modification for Boltzmann exploration called Mellow-max. It, basically, provides an adaptive temperature for Boltzmann exploration.
  
  Here is the link for the paper for tuning mellow-max with deep reinforcement learning (DQN is often mentioned): <http://cs.brown.edu/people/gdk/pubs/tuning_mellowmax_drlw.pdf>
  
  Here is the link for mellow-max implemented with SARSA (I recommend reading this first, to get an understanding of mellow-max): <https://arxiv.org/pdf/1612.05628.pdf>
  
  
  ---
  
  