Answers.Body: <p>This "decay" of later values is a direct consequence of the episodic formula for the objective function for REINFORCE:</p>
  
  <p><span class="math-container">$$J(\theta) = v_{\pi_\theta}(s_0)$$</span></p>
  
  <p>That is, the expected return from the first state of the episode. This is equation 13.4 in the book edition that you linked in the question.</p>
  
  <p>In other words, if there is any discounting, we care less about rewards seen later in the episode. We mainly care about how well the agent will do from its starting position.</p>
  
  <p>This is not true for all formulations of policy gradients. There are other, related, choices of objective function. We can formulate the objective function as caring about the returns from any distribution of states, but in order to define it well, we do need to describe the weighting/distribution somehow, it should be relevant to the problem, and we want to be able to get approximate samples of <span class="math-container">$\nabla J(\theta)$</span> for policy gradient to work. The algorithm you are asking about is specifically for improving policy for episodic problems. Note you can set <span class="math-container">$\gamma = 1$</span> for these problems, so the decay is not necessarily required.</p>
  
  <p>As an aside (because someone is bound to ask): Defining <span class="math-container">$J(\theta)$</span> with respect to all states equally weighted could lead to difficulties  e.g. the objective would take less account of a policy's ability to avoid undesirable states, and it would require a lot of samples from probably irrelevant states in order to estimate it. These difficulties would turn up as a hard  to calculate (or maybe impossible) expectation for <span class="math-container">$\nabla J(\theta)$</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-12T17:37:46.687Z
Answers.LastActivityDate: 2019-02-12T17:43:25.973Z
Answers.LastEditDate: 2019-02-12T17:43:25.973Z
Answers.UserDisplayName: Neil Slater
Body: <p><a href="http://The%20Reinforcement%20Learning%20Book%20by%20Richard%20Sutton%20et%20al" rel="nofollow noreferrer">The Reinforcement Learning Book by Richard Sutton et al, section 13.5</a> shows an online actor critic algorithm.</p>
  
  <p><a href="https://i.sstatic.net/sOtRd.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/sOtRd.png" alt=""></a></p>
  
  <p>Why do the weights updates depend on the discount factor via <span class="math-container">$I$</span>?</p>
  
  <p>It seems that the more we get closer to the end of the episode, the less we value our newest experience <span class="math-container">$\delta$</span>.</p>
  
  <p>This seems odd to me. I thought discounting in the  recursive formula of <span class="math-container">$\delta$</span> itself is enough.</p>
  
  <p><strong>Why does the weights update become less significant as the episode progresses?</strong>
  Note this is not eligibility traces, as those are discussed separately, later in the same episode.</p>
  
Tags: <machine-learning><reinforcement-learning><discount-factor><actor-critic-methods>
Title: In online one step actor critic, why does the weights update become less significant as the episode progresses?
UserDisplayName: Gulzar
ThreadText: # In online one step actor critic, why does the weights update become less significant as the episode progresses?
  
  (Asked by: username_0 on 2019-02-12)
  
  [The Reinforcement Learning Book by Richard Sutton et al, section 13.5](http://The%20Reinforcement%20Learning%20Book%20by%20Richard%20Sutton%20et%20al) shows an online actor critic algorithm.
  
  [![](https://i.sstatic.net/sOtRd.png)](https://i.sstatic.net/sOtRd.png)
  
  Why do the weights updates depend on the discount factor via $I$?
  
  It seems that the more we get closer to the end of the episode, the less we value our newest experience $\delta$.
  
  This seems odd to me. I thought discounting in the recursive formula of $\delta$ itself is enough.
  
  **Why does the weights update become less significant as the episode progresses?**
  Note this is not eligibility traces, as those are discussed separately, later in the same episode.
  
  **Tags:** machine-learning, reinforcement-learning, discount-factor, actor-critic-methods
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-02-12. Score: 1 | Accepted answer
  
  This "decay" of later values is a direct consequence of the episodic formula for the objective function for REINFORCE:
  
  $$J(\theta) = v\_{\pi\_\theta}(s\_0)$$
  
  That is, the expected return from the first state of the episode. This is equation 13.4 in the book edition that you linked in the question.
  
  In other words, if there is any discounting, we care less about rewards seen later in the episode. We mainly care about how well the agent will do from its starting position.
  
  This is not true for all formulations of policy gradients. There are other, related, choices of objective function. We can formulate the objective function as caring about the returns from any distribution of states, but in order to define it well, we do need to describe the weighting/distribution somehow, it should be relevant to the problem, and we want to be able to get approximate samples of $\nabla J(\theta)$ for policy gradient to work. The algorithm you are asking about is specifically for improving policy for episodic problems. Note you can set $\gamma = 1$ for these problems, so the decay is not necessarily required.
  
  As an aside (because someone is bound to ask): Defining $J(\theta)$ with respect to all states equally weighted could lead to difficulties e.g. the objective would take less account of a policy's ability to avoid undesirable states, and it would require a lot of samples from probably irrelevant states in order to estimate it. These difficulties would turn up as a hard to calculate (or maybe impossible) expectation for $\nabla J(\theta)$
  
  
  ---
  
  