Answers.Body: <blockquote>
  <p>So why is <em>constant-<span class="math-container">$\alpha$</span></em> being used?</p>
  </blockquote>
  <p>This is because control scenarios are inherently non-stationary with respect to value functions. Decaying alpha comes with a risk that improvements to the policy will occur progressively more slowly, because the impact to changing the policy will be learned slowly.</p>
  <blockquote>
  <p>From my understanding, in stationary environments, transition probability distribution and reward distribution don't vary with time.</p>
  </blockquote>
  <p>This is correct when considering immediate reward and transitions from any give <span class="math-container">$(s,a)$</span> pair. However you are forgetting that the policy function <span class="math-container">$\pi(a|s)$</span> <em>does</em> vary with time whilst the agent is discovering the optimal policy, and this affects trajetories and expected values that are being estimated.</p>
  <blockquote>
  <p><strong>Question</strong>: How does <span class="math-container">$\alpha$</span> vary in stationary and non-stationary environments?</p>
  </blockquote>
  <p>For non-stationary environments, you will want to maintain some minimum learning rate <span class="math-container">$\alpha$</span> and also some mininum exploration (<span class="math-container">$\epsilon$</span> if you are using <span class="math-container">$\epsilon$</span>-greedy exploration).</p>
  <p>In stationary environments, then a learning rate schedule is still valid and may be useful. You will see it discussed in passing when referencing proofs of convergence for the basic algorithms.</p>
  <p>For example in the <a href="http://incompleteideas.net/book/RLbook2020.pdf" rel="nofollow noreferrer">second edition of Reinforcement Learning: An Introduction</a> it says this regarding convergence of Q learning:</p>
  <blockquote>
  <p>Under this assumption and a variant of the usual stochastic approximation conditions on
  the sequence of step-size parameters, Q has been shown to converge with probability 1 to
  <span class="math-container">$q_*$</span>.</p>
  </blockquote>
  <p>The &quot;usual stochastic approximation conditions on the sequence of step-size parameters&quot; part is a reference to decaying the learning rate.</p>
  <p>However, due to the added complexity of handling exploration vs exploitation, the inherently non-stationary nature of value predictions in control scenarios, and the difficulty of getting function approximation to work, discussions of learning rate decay is a minor detail in texts like Sutton &amp; Barto.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-11T17:56:19.937Z
Answers.Text: I don't follow how "this affects trajectories and expected values that are being estimated" relate to non-stationary environment. Regardless of what policy I follow, for a particular state-action pair,  won't $R(s, a, s')$ be sampled from the same distribution (albeit, varying rewards)?
Answers.UserDisplayName: user529295
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-11T18:29:19.057Z
Answers.Text: @user529295 Yes, but the total return is dependent on the whole trajectory, which in turn depends on the policy, which is changing. Hence non-stationary *expected returns* (not individual expected rewards) even if the environment is itself stationary
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-11T18:34:13.283Z
Answers.Text: Ah! Makes sense.
Answers.UserDisplayName: user529295
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-11T17:04:05.823Z
Answers.LastActivityDate: 2021-08-11T17:16:08.827Z
Answers.LastEditDate: 2021-08-11T17:16:08.827Z
Answers.UserDisplayName: Neil Slater
Body: <p>In Sutton and Barto's book (Chapter 6: TD learning, 2nd edition), he mentions two ways of updating value function:</p>
  <ol>
  <li>Monte Carlo method: <span class="math-container">$V(S_t) \leftarrow V(S_t) + \alpha[G_t - V(S_t)]$</span>.</li>
  <li>TD(0) method: <span class="math-container">$V(S_t) \leftarrow V(S_t) + \alpha[R_{t+1} + \gamma V(S_{t+1}) - V(S_t)]$</span>.</li>
  </ol>
  <p>I understand that <span class="math-container">$\alpha$</span> acts like a learning rate where it take some proportion of MC/TD error and update value function.</p>
  <p>From my understanding, in stationary environments, transition probability distribution and reward distribution don't vary with time. Hence, one should supposedly use <span class="math-container">$\alpha-$</span>decay to update value functions. On the other hand, since distributions change with time in non-stationary environments, <span class="math-container">$\alpha$</span> should be kept constant so as to keep updating the value function with recent TD/MC errors (in other words, history doesn't matter).</p>
  <p>What's been bothering me is that in Example 6.2, 6.5, and 6.7, probability and reward distribution doesn't change. So why is <em>constant-<span class="math-container">$\alpha$</span></em> being used?</p>
  <p><strong>Question</strong>: How does <span class="math-container">$\alpha$</span> vary in stationary and non-stationary environments?</p>
  
Tags: <monte-carlo-methods><temporal-difference-methods><environment><learning-rate>
Title: How does the learning rate $\alpha$ vary in stationary and non-stationary environments?
UserDisplayName: user529295
ThreadText: # How does the learning rate $\alpha$ vary in stationary and non-stationary environments?
  
  (Asked by: username_0 on 2021-08-11)
  
  In Sutton and Barto's book (Chapter 6: TD learning, 2nd edition), he mentions two ways of updating value function:
  
  1. Monte Carlo method: $V(S\_t) \leftarrow V(S\_t) + \alpha[G\_t - V(S\_t)]$.
  2. TD(0) method: $V(S\_t) \leftarrow V(S\_t) + \alpha[R\_{t+1} + \gamma V(S\_{t+1}) - V(S\_t)]$.
  
  I understand that $\alpha$ acts like a learning rate where it take some proportion of MC/TD error and update value function.
  
  From my understanding, in stationary environments, transition probability distribution and reward distribution don't vary with time. Hence, one should supposedly use $\alpha-$decay to update value functions. On the other hand, since distributions change with time in non-stationary environments, $\alpha$ should be kept constant so as to keep updating the value function with recent TD/MC errors (in other words, history doesn't matter).
  
  What's been bothering me is that in Example 6.2, 6.5, and 6.7, probability and reward distribution doesn't change. So why is *constant-$\alpha$* being used?
  
  **Question**: How does $\alpha$ vary in stationary and non-stationary environments?
  
  **Tags:** monte-carlo-methods, temporal-difference-methods, environment, learning-rate
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-08-11. Score: 2 | Accepted answer
  
  > So why is *constant-$\alpha$* being used?
  
  This is because control scenarios are inherently non-stationary with respect to value functions. Decaying alpha comes with a risk that improvements to the policy will occur progressively more slowly, because the impact to changing the policy will be learned slowly.
  
  > From my understanding, in stationary environments, transition probability distribution and reward distribution don't vary with time.
  
  This is correct when considering immediate reward and transitions from any give $(s,a)$ pair. However you are forgetting that the policy function $\pi(a|s)$ *does* vary with time whilst the agent is discovering the optimal policy, and this affects trajetories and expected values that are being estimated.
  
  > **Question**: How does $\alpha$ vary in stationary and non-stationary environments?
  
  For non-stationary environments, you will want to maintain some minimum learning rate $\alpha$ and also some mininum exploration ($\epsilon$ if you are using $\epsilon$-greedy exploration).
  
  In stationary environments, then a learning rate schedule is still valid and may be useful. You will see it discussed in passing when referencing proofs of convergence for the basic algorithms.
  
  For example in the [second edition of Reinforcement Learning: An Introduction](http://incompleteideas.net/book/RLbook2020.pdf) it says this regarding convergence of Q learning:
  
  > Under this assumption and a variant of the usual stochastic approximation conditions on
  > the sequence of step-size parameters, Q has been shown to converge with probability 1 to
  > $q\_\*$.
  
  The "usual stochastic approximation conditions on the sequence of step-size parameters" part is a reference to decaying the learning rate.
  
  However, due to the added complexity of handling exploration vs exploitation, the inherently non-stationary nature of value predictions in control scenarios, and the difficulty of getting function approximation to work, discussions of learning rate decay is a minor detail in texts like Sutton & Barto.
  
  > **Comments:**
  > **username_0**: I don't follow how "this affects trajectories and expected values that are being estimated" relate to non-stationary environment. Regardless of what policy I follow, for a particular state-action pair,  won't $R(s, a, s')$ be sampled from the same distribution (albeit, varying rewards)?  
  > **username_1**: @username_0 Yes, but the total return is dependent on the whole trajectory, which in turn depends on the policy, which is changing. Hence non-stationary *expected returns* (not individual expected rewards) even if the environment is itself stationary  
  > **username_0**: Ah! Makes sense.  
  
  ---
  
  