Answers.Body: <p>Discount factor in (tabular) RL including Q-learning generally acts as a regularization hyperparameter to trade-off optimality with sample efficiency especially for continuous tasks with infinite time horizon. Increasing <span class="math-container">$\gamma$</span> close to <span class="math-container">$1$</span> will evaluate each of agent's actions based on the convergent sum of almost <em>all</em> of its future rewards which obviously results in longer learning computation time. In practice like <span class="math-container">$TD(\lambda)$</span> oftentimes you need to find a balanced discount factor between <span class="math-container">$0$</span> and <span class="math-container">$1$</span> per above mentioned trade-off.</p>
  <p>On the other hand it's a common knowledge in RL with large action space necessarily employing function approximation for control, discounted rewards RL formulation is not even an optimization problem as referenced in papers such as <a href="https://link.springer.com/article/10.1007/BF00114727" rel="nofollow noreferrer"><em>Average reward reinforcement learning: Foundations, algorithms, and empirical results</em></a> by Mahadevan (1996), and <a href="https://arxiv.org/pdf/1910.02140.pdf" rel="nofollow noreferrer"><em>Discounted Reinforcement Learning Is Not an Optimization Problem</em></a> by Sutton et al (2019).</p>
  <blockquote>
  <p>Discounted reinforcement learning is fundamentally incompatible with function
  approximation for control in continuing tasks. It is not an optimization problem in
  its usual formulation, so when using function approximation there is no optimal
  policy. We substantiate these claims, then go on to address some misconceptions
  about discounting and its connection to the average reward formulation. We encourage researchers to adopt rigorous optimization approaches, such as maximizing average reward, for reinforcement learning in continuing tasks</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-12-14T21:00:08.520Z
Answers.LastActivityDate: 2022-12-14T21:00:08.520Z
Answers.UserDisplayName: cinch
Body: <p>While playing around with the learning rate and discount factor in the Q-learning algorithm, I noticed some behavior that I could not really understand myself.</p>
  <p>Firstly, I noticed that increasing the learning rate increased the variance in total reward from the optimal policy. A low learning rate always gave the same total rewards when I ran the algorithm, but a high learning rate yielded different total rewards. Why is that?
  I could see this same behavior when decreasing the discount factor. I once again do not understand why.</p>
  <p>Secondly, I saw that increasing the discount factor resulted in longer learning time. I realize why increasing the learning rate increases learning time (number of episodes until convergence) by definition, but not necessarily why this would hold for the discount factor as well.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-12-31T10:28:51.130Z
Comments.Text: What's your question here? Are you asking for general guidelines on the behaviour of Q-learning changes as you change these hyper-parameters? Please, rewrite the title so that it's your actual question. Someone below provided an answer that talks about the discount factor, but, actually, you didn't ask any question about the discount factor. You're just describing what you observed or your problem.
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><q-learning><learning-rate><discount-factor>
Title: Effects of hyperparameters in Q-learning
UserDisplayName: perceptronEnthusiast420
ThreadText: # Effects of hyperparameters in Q-learning
  
  (Asked by: username_0 on 2022-12-13)
  
  While playing around with the learning rate and discount factor in the Q-learning algorithm, I noticed some behavior that I could not really understand myself.
  
  Firstly, I noticed that increasing the learning rate increased the variance in total reward from the optimal policy. A low learning rate always gave the same total rewards when I ran the algorithm, but a high learning rate yielded different total rewards. Why is that?
  I could see this same behavior when decreasing the discount factor. I once again do not understand why.
  
  Secondly, I saw that increasing the discount factor resulted in longer learning time. I realize why increasing the learning rate increases learning time (number of episodes until convergence) by definition, but not necessarily why this would hold for the discount factor as well.
  
  **Tags:** reinforcement-learning, q-learning, learning-rate, discount-factor
  > **Comments:**
  > **username_1**: What's your question here? Are you asking for general guidelines on the behaviour of Q-learning changes as you change these hyper-parameters? Please, rewrite the title so that it's your actual question. Someone below provided an answer that talks about the discount factor, but, actually, you didn't ask any question about the discount factor. You're just describing what you observed or your problem.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-12-14. Score: 1
  
  Discount factor in (tabular) RL including Q-learning generally acts as a regularization hyperparameter to trade-off optimality with sample efficiency especially for continuous tasks with infinite time horizon. Increasing $\gamma$ close to $1$ will evaluate each of agent's actions based on the convergent sum of almost *all* of its future rewards which obviously results in longer learning computation time. In practice like $TD(\lambda)$ oftentimes you need to find a balanced discount factor between $0$ and $1$ per above mentioned trade-off.
  
  On the other hand it's a common knowledge in RL with large action space necessarily employing function approximation for control, discounted rewards RL formulation is not even an optimization problem as referenced in papers such as [*Average reward reinforcement learning: Foundations, algorithms, and empirical results*](https://link.springer.com/article/10.1007/BF00114727) by Mahadevan (1996), and [*Discounted Reinforcement Learning Is Not an Optimization Problem*](https://arxiv.org/pdf/1910.02140.pdf) by Sutton et al (2019).
  
  > Discounted reinforcement learning is fundamentally incompatible with function
  > approximation for control in continuing tasks. It is not an optimization problem in
  > its usual formulation, so when using function approximation there is no optimal
  > policy. We substantiate these claims, then go on to address some misconceptions
  > about discounting and its connection to the average reward formulation. We encourage researchers to adopt rigorous optimization approaches, such as maximizing average reward, for reinforcement learning in continuing tasks
  
  
  ---
  
  