Answers.Body: <p>In RL, there are episodic and non-episodic tasks (or problems). In episodic tasks, each episode proceeds in time steps.</p>
  
  <p>For example, most games are episodic tasks. For instance, in a football championship (e.g. Premier League), each football match during the whole season is an episode. In this example, each minute (or second) of a football match can be considered a time step of the the episode (that is, the football match).</p>
  
  <p>The parameter <code>total_episodes</code> thus specifies the number of episodes of your RL episodic problem. Similarly, <code>max_steps</code> specifies the maximum number of time steps per episode. </p>
  
  <p>Why do we also need <code>total_test_episodes</code>? In machine learning, when building a model, there are usually two phases: the training phase and the test phases. In the training phase, you use the training dataset to learn the parameters of the model. In the test phase, you test the performance (e.g. the total return, in the case of RL) of the model. Hence, <code>total_test_episodes</code> is used to specify this hyper-parameter.</p>
  
  <p>In deep RL, a (deep) neural network (NN) is used to represent either a value function or a policy (which is also a function). In machine learning, neural networks are usually trained using an <a href="https://ai.stackexchange.com/a/8985/2444">optimisation method, like gradient descent (GD) and the back-propagation (BP), which is used to compute the gradient of the objective function with respect to the parameters of the model</a>. In the case of deep RL, the parameters of the NN that represents either the value function or policy are also learned using a similar approach. In this context, the learning rate is a hyper-parameter that determines the "strength" of the update step of the optimisation algorithm. More concretely, in the case of GD, the update step is</p>
  
  <p><span class="math-container">$$\mathbf{\theta}_{n+1} \gets \mathbf{\theta}_{n} - \alpha \nabla f(\mathbf{\theta} _{n})$$</span></p>
  
  <p>where <span class="math-container">$\mathbf{\theta}_{n+1}$</span> is a vector containing the parameter of your model (in this case, a NN), <span class="math-container">$\alpha$</span> is the learning rate and <span class="math-container">$\nabla f(\mathbf{\theta} _{n})$</span> is the gradient of the objective function <span class="math-container">$f$</span>. Hence, <code>learning_rate</code> specifies the value of <span class="math-container">$\alpha$</span>. </p>
  
  <p>In your case, <code>learning_rate</code> can actually specify the value of the learning rate of your RL algorithm. For example, in the case <a href="https://ai.stackexchange.com/a/10560/2444"><span class="math-container">$Q$</span>-learning</a>, the learning rate, which we can denote by <span class="math-container">$\alpha$</span>, also specifies the "strength" of the update.</p>
  
  <p>Similarly, the <code>gamma</code> is a hyper-parameter of your RL algorithm. For example, in the case <span class="math-container">$Q$</span>-learning, the parameter <span class="math-container">$\gamma$</span> (the discount factor) which determines the contribution of the estimate of the <span class="math-container">$Q$</span> value of the next state (while taking the greedy action from that state) to the <span class="math-container">$Q$</span> value that is being currently updated. </p>
  
  <p><span class="math-container">$Q$</span>-learning is an off-policy RL algorithm, which means that it uses a behaviour policy that is possibly different than the policy it tries to estimate. The usual behaviour policy is the <span class="math-container">$\epsilon$</span>-greedy (with probability <span class="math-container">$\epsilon$</span> a random action is taken in a certain state and with probability <span class="math-container">$1 - \epsilon$</span> the greedy action is taken). The parameter <code>epsilon</code> specifies this hyper-parameter. In this context, an initial <span class="math-container">$\epsilon$</span> that specifies the exploration rate is provided. As the training progresses, the estimate of the optimal value function should become more accurate. In that case, you want to explore less and follow more your estimate of the optimal value function, so <code>min_epsilon</code> is used to specify the lowest <span class="math-container">$\epsilon$</span>.</p>
  
  <p>The parameter <code>decay_rate</code> is used to specify the value of the decay rate. Have a look at <a href="https://stats.stackexchange.com/a/31334/82135">https://stats.stackexchange.com/a/31334/82135</a> for more info.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-13T16:51:47.997Z
Answers.Text: @nbro Isn't gamma related to the importance you want to give to the value of the rewards in the next states? Instead of, as you said, the future values of the Q value? Or maybe that was what you wanted to say and I misunderstood it.
Answers.UserDisplayName: Miguel Saraiva
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-13T19:51:00.657Z
Answers.Text: @MiguelSaraiva Yes, I think I stated it wrongly. It is the value in the next state.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-13T09:35:01.510Z
Answers.LastActivityDate: 2019-06-14T09:38:12.373Z
Answers.LastEditDate: 2019-06-14T09:38:12.373Z
Answers.UserDisplayName: nbro
Body: <p>I am new in RL and I am trying to understand why do we need all these hyperparameters.
  Can somebody explain me why we use them and what are the best values to use for them?</p>
  
  <blockquote>
    <p>total_episodes = 50000        # Total episodes</p>
    
    <p>total_test_episodes = 100     # Total test episodes</p>
    
    <p>max_steps = 99                # Max steps per episode</p>
    
    <p>learning_rate = 0.7           # Learning rate</p>
    
    <p>gamma = 0.618                 # Discounting rate</p>
    
    <h1>Exploration parameters</h1>
    
    <p>epsilon = 1.0                 # Exploration rate</p>
    
    <p>max_epsilon = 1.0             # Exploration probability at start</p>
    
    <p>min_epsilon = 0.01            # Minimum exploration probability</p>
    
    <p>decay_rate = 0.01             # Exponential decay rate</p>
  </blockquote>
  
  <p>I am currently working on taxi_v2 problem from GYM.</p>
  
  <p>Link: <a href="https://learndatasci.com/tutorials/reinforcement-q-learning-scratch-python-openai-gym/" rel="nofollow noreferrer">https://learndatasci.com/tutorials/reinforcement-q-learning-scratch-python-openai-gym/</a></p>
  
Tags: <machine-learning><reinforcement-learning><deep-rl>
Title: Why are we using all hyperparameters in RL?
UserDisplayName: justStarting
ThreadText: # Why are we using all hyperparameters in RL?
  
  (Asked by: username_0 on 2019-06-13)
  
  I am new in RL and I am trying to understand why do we need all these hyperparameters.
  Can somebody explain me why we use them and what are the best values to use for them?
  
  > total\_episodes = 50000 # Total episodes
  >
  > total\_test\_episodes = 100 # Total test episodes
  >
  > max\_steps = 99 # Max steps per episode
  >
  > learning\_rate = 0.7 # Learning rate
  >
  > gamma = 0.618 # Discounting rate
  >
  > # Exploration parameters
  >
  > epsilon = 1.0 # Exploration rate
  >
  > max\_epsilon = 1.0 # Exploration probability at start
  >
  > min\_epsilon = 0.01 # Minimum exploration probability
  >
  > decay\_rate = 0.01 # Exponential decay rate
  
  I am currently working on taxi\_v2 problem from GYM.
  
  Link: <https://learndatasci.com/tutorials/reinforcement-q-learning-scratch-python-openai-gym/>
  
  **Tags:** machine-learning, reinforcement-learning, deep-rl
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-06-13. Score: 1 | Accepted answer
  
  In RL, there are episodic and non-episodic tasks (or problems). In episodic tasks, each episode proceeds in time steps.
  
  For example, most games are episodic tasks. For instance, in a football championship (e.g. Premier League), each football match during the whole season is an episode. In this example, each minute (or second) of a football match can be considered a time step of the the episode (that is, the football match).
  
  The parameter `total_episodes` thus specifies the number of episodes of your RL episodic problem. Similarly, `max_steps` specifies the maximum number of time steps per episode.
  
  Why do we also need `total_test_episodes`? In machine learning, when building a model, there are usually two phases: the training phase and the test phases. In the training phase, you use the training dataset to learn the parameters of the model. In the test phase, you test the performance (e.g. the total return, in the case of RL) of the model. Hence, `total_test_episodes` is used to specify this hyper-parameter.
  
  In deep RL, a (deep) neural network (NN) is used to represent either a value function or a policy (which is also a function). In machine learning, neural networks are usually trained using an [optimisation method, like gradient descent (GD) and the back-propagation (BP), which is used to compute the gradient of the objective function with respect to the parameters of the model](https://ai.stackexchange.com/a/8985/2444). In the case of deep RL, the parameters of the NN that represents either the value function or policy are also learned using a similar approach. In this context, the learning rate is a hyper-parameter that determines the "strength" of the update step of the optimisation algorithm. More concretely, in the case of GD, the update step is
  
  $$\mathbf{\theta}\_{n+1} \gets \mathbf{\theta}\_{n} - \alpha \nabla f(\mathbf{\theta} \_{n})$$
  
  where $\mathbf{\theta}\_{n+1}$ is a vector containing the parameter of your model (in this case, a NN), $\alpha$ is the learning rate and $\nabla f(\mathbf{\theta} \_{n})$ is the gradient of the objective function $f$. Hence, `learning_rate` specifies the value of $\alpha$.
  
  In your case, `learning_rate` can actually specify the value of the learning rate of your RL algorithm. For example, in the case [$Q$-learning](https://ai.stackexchange.com/a/10560/2444), the learning rate, which we can denote by $\alpha$, also specifies the "strength" of the update.
  
  Similarly, the `gamma` is a hyper-parameter of your RL algorithm. For example, in the case $Q$-learning, the parameter $\gamma$ (the discount factor) which determines the contribution of the estimate of the $Q$ value of the next state (while taking the greedy action from that state) to the $Q$ value that is being currently updated.
  
  $Q$-learning is an off-policy RL algorithm, which means that it uses a behaviour policy that is possibly different than the policy it tries to estimate. The usual behaviour policy is the $\epsilon$-greedy (with probability $\epsilon$ a random action is taken in a certain state and with probability $1 - \epsilon$ the greedy action is taken). The parameter `epsilon` specifies this hyper-parameter. In this context, an initial $\epsilon$ that specifies the exploration rate is provided. As the training progresses, the estimate of the optimal value function should become more accurate. In that case, you want to explore less and follow more your estimate of the optimal value function, so `min_epsilon` is used to specify the lowest $\epsilon$.
  
  The parameter `decay_rate` is used to specify the value of the decay rate. Have a look at <https://stats.stackexchange.com/a/31334/82135> for more info.
  
  > **Comments:**
  > **username_2**: @username_1 Isn't gamma related to the importance you want to give to the value of the rewards in the next states? Instead of, as you said, the future values of the Q value? Or maybe that was what you wanted to say and I misunderstood it.  
  > **username_1**: @username_2 Yes, I think I stated it wrongly. It is the value in the next state.  
  
  ---
  
  