Answers.Body: <p>In general, NO.</p>
  <p>You don't get the &quot;true&quot; state value function.  TD-learning approximates the true value function. It can be a very close, or even exact approximation in simple cases, but, in general, it is just an approximation.</p>
  <p>Depending on the difficulty of the problem, a non-constant <span class="math-container">$\alpha$</span> value can help the policy approximate the true value function more quickly, or help the learning from getting stuck in a local minimum.</p>
  <p>There are implementations, like ADAM, which will adaptively change the learning rate for each feature.</p>
  <p>Usually, you can expect convergence must faster if you adaptively change the learning rate by using an implementation like ADAM (see <a href="https://www.fast.ai/2018/07/02/adam-weight-decay/" rel="nofollow noreferrer">this</a>).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-16T12:50:50.260Z
Answers.Text: So why can't it get the "true" value function? Could you perhaps show it mathematically?
Answers.UserDisplayName: XXX
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-17T03:00:54.467Z
Answers.Text: Most of the time the "true" value function is not known, and cannot be known.   We then use iterate learning to try to approximate this value function. A good read can be found here -> https://machinelearningmastery.com/neural-networks-are-function-approximators/.  Also sorry nbro, i meant 'non-constanct', i've fixed the post.
Answers.UserDisplayName: John Rothman
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-16T12:47:51.880Z
Answers.LastActivityDate: 2021-10-17T20:53:26.727Z
Answers.LastEditDate: 2021-10-17T20:53:26.727Z
Answers.UserDisplayName: John Rothman
Body: <p>In Temporal-Difference Learning, we update our value function by <span class="math-container">$V\left(S_{t}\right) \leftarrow V\left(S_{t}\right)+\alpha\left(R_{t+1}+\gamma V\left(S_{t+1}\right)-V\left(S_{t}\right)\right)$</span></p>
  <p>If we choose a constant <span class="math-container">$\alpha$</span>, will the algorithm eventually give us the true state value function? Why or why not?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-10-16T13:44:01.923Z
Comments.Text: There are multiple TD-learning algorithms (e.g. Q-learning, SARSA, or even just policy evaluation algorithms, which is what you seem to be referring to). It may be a good idea to clarify what you're exactly referring to (e.g. provide the link to or name of section/chapter of book that describes the algorithm you have in mind: for example, this one [https://incompleteideas.net/book/RLbook2020.pdf#page=142](https://incompleteideas.net/book/RLbook2020.pdf#page=142)), so that people do not misinterpret your question/context.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-10-17T07:42:53.973Z
Comments.Text: @nbro: This looks like single-step TD learning for sample-based policy evaluation, not a specific optimal control algorithm (and not dynamic programming). This is the core of many model-free prediction and control algorithms, and described in S&B chapter 6 sections 1 to 3. It would help if the OP did add some scope to the question, since in simple, tabular environments it is possible to guarantee exact convergence even with $\alpha = 1$ - whilst the answer so far focuses on function approximation, it is not clear if that is important to OP.
Comments.UserDisplayName: Neil Slater
Tags: <reinforcement-learning><reference-request><convergence><temporal-difference-methods><learning-rate>
Title: How does $\alpha$ affect the convergence of the TD algorithm?
UserDisplayName: XXX
ThreadText: # How does $\alpha$ affect the convergence of the TD algorithm?
  
  (Asked by: username_0 on 2021-10-16)
  
  In Temporal-Difference Learning, we update our value function by $V\left(S\_{t}\right) \leftarrow V\left(S\_{t}\right)+\alpha\left(R\_{t+1}+\gamma V\left(S\_{t+1}\right)-V\left(S\_{t}\right)\right)$
  
  If we choose a constant $\alpha$, will the algorithm eventually give us the true state value function? Why or why not?
  
  **Tags:** reinforcement-learning, reference-request, convergence, temporal-difference-methods, learning-rate
  > **Comments:**
  > **username_1**: There are multiple TD-learning algorithms (e.g. Q-learning, SARSA, or even just policy evaluation algorithms, which is what you seem to be referring to). It may be a good idea to clarify what you're exactly referring to (e.g. provide the link to or name of section/chapter of book that describes the algorithm you have in mind: for example, this one [https://incompleteideas.net/book/RLbook2020.pdf#page=142](https://incompleteideas.net/book/RLbook2020.pdf#page=142)), so that people do not misinterpret your question/context.  
  > **username_2**: @nbro: This looks like single-step TD learning for sample-based policy evaluation, not a specific optimal control algorithm (and not dynamic programming). This is the core of many model-free prediction and control algorithms, and described in S&B chapter 6 sections 1 to 3. It would help if the OP did add some scope to the question, since in simple, tabular environments it is possible to guarantee exact convergence even with $\alpha = 1$ - whilst the answer so far focuses on function approximation, it is not clear if that is important to OP.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2021-10-16. Score: 0
  
  In general, NO.
  
  You don't get the "true" state value function. TD-learning approximates the true value function. It can be a very close, or even exact approximation in simple cases, but, in general, it is just an approximation.
  
  Depending on the difficulty of the problem, a non-constant $\alpha$ value can help the policy approximate the true value function more quickly, or help the learning from getting stuck in a local minimum.
  
  There are implementations, like ADAM, which will adaptively change the learning rate for each feature.
  
  Usually, you can expect convergence must faster if you adaptively change the learning rate by using an implementation like ADAM (see [this](https://www.fast.ai/2018/07/02/adam-weight-decay/)).
  
  > **Comments:**
  > **username_0**: So why can't it get the "true" value function? Could you perhaps show it mathematically?  
  > **username_3**: Most of the time the "true" value function is not known, and cannot be known.   We then use iterate learning to try to approximate this value function. A good read can be found here -> https://machinelearningmastery.com/neural-networks-are-function-approximators/.  Also sorry nbro, i meant 'non-constanct', i've fixed the post.  
  
  ---
  
  