Answers.Body: <p>I think I understand now. The implementation uses a <a href="https://spinningup.openai.com/en/latest/spinningup/rl_intro3.html" rel="nofollow noreferrer">reward to go</a> version, which can be proved to be equivalent to the original loss, see <a href="https://ai.stackexchange.com/questions/9614/why-does-the-reward-to-go-trick-in-policy-gradient-methods-work">here</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-26T02:28:32.483Z
Answers.LastActivityDate: 2024-01-26T02:28:32.483Z
Answers.UserDisplayName: Chris XU
Answers.Body: <p>The policy gradient objective function of your Berkeley reference is <em>undiscounted</em> value function of a start state in finite horizon episodic cases, while the other Sutton's reference at least for the boxed pseudocode is about <em>discounted</em> value function of a start state in finite horizon episodic cases as stated in the book:</p>
  <blockquote>
  <p>From here on in our discussion we will assume no discounting (<span class="math-container">$\gamma = 1$</span>) for the episodic case, although for completeness we do include the possibility of discounting in the boxed algorithms.</p>
  </blockquote>
  <p>Since the undiscounted total return <span class="math-container">$G_t$</span> is defined as <span class="math-container">$\sum_t^T r_t(s_t,a_t)$</span>, there's no essential notation difference here. Berkeley's equation is an approximation expressed via samples <span class="math-container">$i$</span>, so it may not be easy to use <span class="math-container">$G_t$</span> without further unnecessary confusions such as <span class="math-container">$G_{t,i}$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-27T01:09:44.367Z
Answers.LastActivityDate: 2024-01-27T01:09:44.367Z
Answers.UserDisplayName: cinch
Body: <p>I have a question regarding the implementation of the REINFORCE algorithm.</p>
  <p>In <a href="https://rll.berkeley.edu/deeprlcourse/f17docs/lecture_4_policy_gradient.pdf" rel="nofollow noreferrer">berkeley course</a> (see slide 9) the gradient is defined as
  <a href="https://i.sstatic.net/8uJqT.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/8uJqT.png" alt="enter image description here" /></a></p>
  <p>Note that the return <strong>sums from 1</strong>. However in Sutton's book the return <strong>sums from t+1</strong>
  <a href="https://i.sstatic.net/66LMm.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/66LMm.png" alt="enter image description here" /></a></p>
  <p>In fact in two popular implementations, <a href="https://github.com/pytorch/examples/blob/main/reinforcement_learning/reinforce.py" rel="nofollow noreferrer">pytorch's official example</a> (See Line 71) and <a href="https://github.com/thu-ml/tianshou/blob/master/tianshou/policy/modelfree/pg.py#L226" rel="nofollow noreferrer">tianshou's example</a> (See Line 226), if I understand correctly, they are computing</p>
  <pre><code>sum_t (∇log π(a_t | s_t) * R_t)
  </code></pre>
  <p>which seems to correspond to Sutton's definition rather than Berkeley's (which will becomes</p>
  <pre><code>(sum_t ∇log π(a_t | s_t)) * (sum_t r_t)
  </code></pre>
  <p>).</p>
  <p>I wonder which definition is the de facto standard in the community. Thank you.</p>
  
Tags: <reinforcement-learning><deep-rl><reinforce>
Title: Implementation difference in REINFORCE algorithm, where to sum from
UserDisplayName: Chris XU
ThreadText: # Implementation difference in REINFORCE algorithm, where to sum from
  
  (Asked by: username_0 on 2024-01-25)
  
  I have a question regarding the implementation of the REINFORCE algorithm.
  In berkeley course (see slide 9) the gradient is defined as
  
  Note that the return sums from 1. However in Sutton's book the return sums from t+1
  
  In fact in two popular implementations, pytorch's official example (See Line 71) and tianshou's example (See Line 226), if I understand correctly, they are computing
  sum_t (∇log π(a_t | s_t) * R_t)
  
  which seems to correspond to Sutton's definition rather than Berkeley's (which will becomes
  (sum_t ∇log π(a_t | s_t)) * (sum_t r_t)
  
  ).
  I wonder which definition is the de facto standard in the community. Thank you.
  
  **Tags:** reinforcement-learning, deep-rl, reinforce
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2024-01-27. Score: 2 | Accepted answer
  
  The policy gradient objective function of your Berkeley reference is *undiscounted* value function of a start state in finite horizon episodic cases, while the other Sutton's reference at least for the boxed pseudocode is about *discounted* value function of a start state in finite horizon episodic cases as stated in the book:
  
  > From here on in our discussion we will assume no discounting ($\gamma = 1$) for the episodic case, although for completeness we do include the possibility of discounting in the boxed algorithms.
  
  Since the undiscounted total return $G\_t$ is defined as $\sum\_t^T r\_t(s\_t,a\_t)$, there's no essential notation difference here. Berkeley's equation is an approximation expressed via samples $i$, so it may not be easy to use $G\_t$ without further unnecessary confusions such as $G\_{t,i}$.
  
  
  ---
  
  ### Answer by username_0 on 2024-01-26. Score: 1
  
  I think I understand now. The implementation uses a [reward to go](https://spinningup.openai.com/en/latest/spinningup/rl_intro3.html) version, which can be proved to be equivalent to the original loss, see [here](https://ai.stackexchange.com/questions/9614/why-does-the-reward-to-go-trick-in-policy-gradient-methods-work).
  
  
  ---
  
  