<blockquote>
<p>So why is <em>constant-<span class="math-container">$\alpha$</span></em> being used?</p>
</blockquote>
<p>This is because control scenarios are inherently non-stationary with respect to value functions. Decaying alpha comes with a risk that improvements to the policy will occur progressively more slowly, because the impact to changing the policy will be learned slowly.</p>
<blockquote>
<p>From my understanding, in stationary environments, transition probability distribution and reward distribution don't vary with time.</p>
</blockquote>
<p>This is correct when considering immediate reward and transitions from any give <span class="math-container">$(s,a)$</span> pair. However you are forgetting that the policy function <span class="math-container">$\pi(a|s)$</span> <em>does</em> vary with time whilst the agent is discovering the optimal policy, and this affects trajetories and expected values that are being estimated.</p>
<blockquote>
<p><strong>Question</strong>: How does <span class="math-container">$\alpha$</span> vary in stationary and non-stationary environments?</p>
</blockquote>
<p>For non-stationary environments, you will want to maintain some minimum learning rate <span class="math-container">$\alpha$</span> and also some mininum exploration (<span class="math-container">$\epsilon$</span> if you are using <span class="math-container">$\epsilon$</span>-greedy exploration).</p>
<p>In stationary environments, then a learning rate schedule is still valid and may be useful. You will see it discussed in passing when referencing proofs of convergence for the basic algorithms.</p>
<p>For example in the <a href="http://incompleteideas.net/book/RLbook2020.pdf" rel="nofollow noreferrer">second edition of Reinforcement Learning: An Introduction</a> it says this regarding convergence of Q learning:</p>
<blockquote>
<p>Under this assumption and a variant of the usual stochastic approximation conditions on
the sequence of step-size parameters, Q has been shown to converge with probability 1 to
<span class="math-container">$q_*$</span>.</p>
</blockquote>
<p>The &quot;usual stochastic approximation conditions on the sequence of step-size parameters&quot; part is a reference to decaying the learning rate.</p>
<p>However, due to the added complexity of handling exploration vs exploitation, the inherently non-stationary nature of value predictions in control scenarios, and the difficulty of getting function approximation to work, discussions of learning rate decay is a minor detail in texts like Sutton &amp; Barto.</p>

CC BY-SA 4.0
2021-08-11T17:56:19.937Z
I don't follow how "this affects trajectories and expected values that are being estimated" relate to non-stationary environment. Regardless of what policy I follow, for a particular state-action pair,  won't $R(s, a, s')$ be sampled from the same distribution (albeit, varying rewards)?
user529295
CC BY-SA 4.0
2021-08-11T18:29:19.057Z
@user529295 Yes, but the total return is dependent on the whole trajectory, which in turn depends on the policy, which is changing. Hence non-stationary *expected returns* (not individual expected rewards) even if the environment is itself stationary
Neil Slater
CC BY-SA 4.0
2021-08-11T18:34:13.283Z
Ah! Makes sense.
user529295
CC BY-SA 4.0
2021-08-11T17:04:05.823Z
2021-08-11T17:16:08.827Z
2021-08-11T17:16:08.827Z
Neil Slater
<p>In Sutton and Barto's book (Chapter 6: TD learning, 2nd edition), he mentions two ways of updating value function:</p>
<ol>
<li>Monte Carlo method: <span class="math-container">$V(S_t) \leftarrow V(S_t) + \alpha[G_t - V(S_t)]$</span>.</li>
<li>TD(0) method: <span class="math-container">$V(S_t) \leftarrow V(S_t) + \alpha[R_{t+1} + \gamma V(S_{t+1}) - V(S_t)]$</span>.</li>
</ol>
<p>I understand that <span class="math-container">$\alpha$</span> acts like a learning rate where it take some proportion of MC/TD error and update value function.</p>
<p>From my understanding, in stationary environments, transition probability distribution and reward distribution don't vary with time. Hence, one should supposedly use <span class="math-container">$\alpha-$</span>decay to update value functions. On the other hand, since distributions change with time in non-stationary environments, <span class="math-container">$\alpha$</span> should be kept constant so as to keep updating the value function with recent TD/MC errors (in other words, history doesn't matter).</p>
<p>What's been bothering me is that in Example 6.2, 6.5, and 6.7, probability and reward distribution doesn't change. So why is <em>constant-<span class="math-container">$\alpha$</span></em> being used?</p>
<p><strong>Question</strong>: How does <span class="math-container">$\alpha$</span> vary in stationary and non-stationary environments?</p>

<monte-carlo-methods><temporal-difference-methods><environment><learning-rate>
How does the learning rate $\alpha$ vary in stationary and non-stationary environments?
user529295
# How does the learning rate $\alpha$ vary in stationary and non-stationary environments?

(Asked by: username_0 on 2021-08-11)

In Sutton and Barto's book (Chapter 6: TD learning, 2nd edition), he mentions two ways of updating value function:

1. Monte Carlo method: $V(S\_t) \leftarrow V(S\_t) + \alpha[G\_t - V(S\_t)]$.
2. TD(0) method: $V(S\_t) \leftarrow V(S\_t) + \alpha[R\_{t+1} + \gamma V(S\_{t+1}) - V(S\_t)]$.

I understand that $\alpha$ acts like a learning rate where it take some proportion of MC/TD error and update value function.

From my understanding, in stationary environments, transition probability distribution and reward distribution don't vary with time. Hence, one should supposedly use $\alpha-$decay to update value functions. On the other hand, since distributions change with time in non-stationary environments, $\alpha$ should be kept constant so as to keep updating the value function with recent TD/MC errors (in other words, history doesn't matter).

What's been bothering me is that in Example 6.2, 6.5, and 6.7, probability and reward distribution doesn't change. So why is *constant-$\alpha$* being used?

**Question**: How does $\alpha$ vary in stationary and non-stationary environments?

**Tags:** monte-carlo-methods, temporal-difference-methods, environment, learning-rate


---

## 1 answers

### Answer by username_1 on 2021-08-11. Score: 2 | Accepted answer

> So why is *constant-$\alpha$* being used?

This is because control scenarios are inherently non-stationary with respect to value functions. Decaying alpha comes with a risk that improvements to the policy will occur progressively more slowly, because the impact to changing the policy will be learned slowly.

> From my understanding, in stationary environments, transition probability distribution and reward distribution don't vary with time.

This is correct when considering immediate reward and transitions from any give $(s,a)$ pair. However you are forgetting that the policy function $\pi(a|s)$ *does* vary with time whilst the agent is discovering the optimal policy, and this affects trajetories and expected values that are being estimated.

> **Question**: How does $\alpha$ vary in stationary and non-stationary environments?

For non-stationary environments, you will want to maintain some minimum learning rate $\alpha$ and also some mininum exploration ($\epsilon$ if you are using $\epsilon$-greedy exploration).

In stationary environments, then a learning rate schedule is still valid and may be useful. You will see it discussed in passing when referencing proofs of convergence for the basic algorithms.

For example in the [second edition of Reinforcement Learning: An Introduction](http://incompleteideas.net/book/RLbook2020.pdf) it says this regarding convergence of Q learning:

> Under this assumption and a variant of the usual stochastic approximation conditions on
> the sequence of step-size parameters, Q has been shown to converge with probability 1 to
> $q\_\*$.

The "usual stochastic approximation conditions on the sequence of step-size parameters" part is a reference to decaying the learning rate.

However, due to the added complexity of handling exploration vs exploitation, the inherently non-stationary nature of value predictions in control scenarios, and the difficulty of getting function approximation to work, discussions of learning rate decay is a minor detail in texts like Sutton & Barto.

> **Comments:**
> **username_0**: I don't follow how "this affects trajectories and expected values that are being estimated" relate to non-stationary environment. Regardless of what policy I follow, for a particular state-action pair,  won't $R(s, a, s')$ be sampled from the same distribution (albeit, varying rewards)?  
> **username_1**: @username_0 Yes, but the total return is dependent on the whole trajectory, which in turn depends on the policy, which is changing. Hence non-stationary *expected returns* (not individual expected rewards) even if the environment is itself stationary  
> **username_0**: Ah! Makes sense.  

---

