Answers.Body: <p>You should read this study <a href="https://arxiv.org/abs/2006.05990" rel="noreferrer">https://arxiv.org/abs/2006.05990</a> which does some empirical study on this question, specifically for on-policy, continuous action space DRL.</p>
  <p>It suggests that discount factor and learning rate are the two most important parameters to tune, followed by the width of the policy/value functions.</p>
  <p>That study also reports that it's very important to normalize the observations, and initialize the policy so that the initial actions are zero mean with a very small variance.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-29T15:08:56.560Z
Answers.LastActivityDate: 2021-05-29T15:08:56.560Z
Answers.UserDisplayName: Taw
Answers.Body: <p>Personally, I would choose the following two as the most important:</p>
  <ul>
  <li><strong>epsilon</strong>: When using an epsilon-greedy policy, epsilon determines how often the agent should <em>explore</em> and how often it should <em>exploit</em>. Balancing exploration and exploitation is crucial for the success of the learning agent. Too little exploration might not teach anything to the agent and too much exploration might just waste your time.</li>
  <li><strong>learning rate</strong>: The learning rate determines how fast do you learn from new states of experience. A learning rate that is too high might not be good in cases when the environment has many states with high probabilities of negative rewards, i.e. many penalizations. This might make your agent move back and forth in the same place in order to avoid getting penalized. Also, a learning rate that is too low might make your agent learn very slowly and depending on your epsilon, the agent might enter a phase of exploitation with very little knowledge of an optimal policy.</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-28T13:53:09.710Z
Answers.LastActivityDate: 2021-05-28T14:42:14.473Z
Answers.LastEditDate: 2021-05-28T14:42:14.473Z
Answers.UserDisplayName: devidduma
Body: <p>Obviously, this is somewhat subjective, but what hyper-parameters typically have the most significant impact on an RL agent's ability to learn? For example, the replay buffer size, learning rate, entropy coefficient, etc.</p>
  <p>For example, in &quot;normal&quot; ML, the batch size and learning rate are typically the main hyper-parameters that get optimised first.</p>
  <p>Specifically, I am using PPO, but this can probably be applied to a lot of other RL algorithms too.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-05-29T00:31:45.923Z
Comments.Text: Maybe rather than asking "What are the **best**..."? Maybe you could reword your question to something like "Which hyper-parameters typically affect the performance of the RL algorithms the most, so that should be tuned first? Are there any guidelines or research on this topic?"
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><deep-rl><hyperparameter-optimization><hyper-parameters><proximal-policy-optimization>
Title: What are the best hyper-parameters to tune in reinforcement learning?
UserDisplayName: Dylan Kerler
ThreadText: # What are the best hyper-parameters to tune in reinforcement learning?
  
  (Asked by: username_0 on 2021-05-28)
  
  Obviously, this is somewhat subjective, but what hyper-parameters typically have the most significant impact on an RL agent's ability to learn? For example, the replay buffer size, learning rate, entropy coefficient, etc.
  
  For example, in "normal" ML, the batch size and learning rate are typically the main hyper-parameters that get optimised first.
  
  Specifically, I am using PPO, but this can probably be applied to a lot of other RL algorithms too.
  
  **Tags:** reinforcement-learning, deep-rl, hyperparameter-optimization, hyper-parameters, proximal-policy-optimization
  > **Comments:**
  > **username_1**: Maybe rather than asking "What are the **best**..."? Maybe you could reword your question to something like "Which hyper-parameters typically affect the performance of the RL algorithms the most, so that should be tuned first? Are there any guidelines or research on this topic?"  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2021-05-29. Score: 6
  
  You should read this study <https://arxiv.org/abs/2006.05990> which does some empirical study on this question, specifically for on-policy, continuous action space DRL.
  
  It suggests that discount factor and learning rate are the two most important parameters to tune, followed by the width of the policy/value functions.
  
  That study also reports that it's very important to normalize the observations, and initialize the policy so that the initial actions are zero mean with a very small variance.
  
  
  ---
  
  ### Answer by username_3 on 2021-05-28. Score: 5
  
  Personally, I would choose the following two as the most important:
  
  * **epsilon**: When using an epsilon-greedy policy, epsilon determines how often the agent should *explore* and how often it should *exploit*. Balancing exploration and exploitation is crucial for the success of the learning agent. Too little exploration might not teach anything to the agent and too much exploration might just waste your time.
  * **learning rate**: The learning rate determines how fast do you learn from new states of experience. A learning rate that is too high might not be good in cases when the environment has many states with high probabilities of negative rewards, i.e. many penalizations. This might make your agent move back and forth in the same place in order to avoid getting penalized. Also, a learning rate that is too low might make your agent learn very slowly and depending on your epsilon, the agent might enter a phase of exploitation with very little knowledge of an optimal policy.
  
  
  ---
  
  