Answers.Body: <p>I think I understand now. The implementation uses a <a href="https://spinningup.openai.com/en/latest/spinningup/rl_intro3.html" rel="nofollow noreferrer">reward to go</a> version, which can be proved to be equivalent to the original loss, see <a href="https://ai.stackexchange.com/questions/9614/why-does-the-reward-to-go-trick-in-policy-gradient-methods-work">here</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-26T02:28:32.483Z
Answers.LastActivityDate: 2024-01-26T02:28:32.483Z
Answers.UserDisplayName: Chris XU
Answers.Body: <p>The policy gradient objective function of your Berkeley reference is <em>undiscounted</em> value function of a start state in finite horizon episodic cases, while the other Sutton's reference at least for the boxed pseudocode is about <em>discounted</em> value function of a start state in finite horizon episodic cases as stated in the book:</p>
  <blockquote>
  <p>From here on in our discussion we will assume no discounting (<span class="math-container">$\gamma = 1$</span>) for the episodic case, although for completeness we do include the possibility of discounting in the boxed algorithms.</p>
  </blockquote>
  <p>Since the undiscounted total return <span class="math-container">$G_t$</span> is defined as <span class="math-container">$\sum_t^T r_t(s_t,a_t)$</span>, there's no essential notation difference here. Berkeley's equation is an approximation expressed via samples <span class="math-container">$i$</span>, so it may not be easy to use <span class="math-container">$G_t$</span> without further unnecessary confusions such as <span class="math-container">$G_{t,i}$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-27T01:09:44.367Z
Answers.LastActivityDate: 2024-01-27T01:09:44.367Z
Answers.UserDisplayName: cinch
Body: <p>I have a question regarding the implementation of the REINFORCE algorithm.</p>
  <p>In <a href="https://rll.berkeley.edu/deeprlcourse/f17docs/lecture_4_policy_gradient.pdf" rel="nofollow noreferrer">berkeley course</a> (see slide 9) the gradient is defined as
  <a href="https://i.sstatic.net/8uJqT.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/8uJqT.png" alt="enter image description here" /></a></p>
  <p>Note that the return <strong>sums from 1</strong>. However in Sutton's book the return <strong>sums from t+1</strong>
  <a href="https://i.sstatic.net/66LMm.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/66LMm.png" alt="enter image description here" /></a></p>
  <p>In fact in two popular implementations, <a href="https://github.com/pytorch/examples/blob/main/reinforcement_learning/reinforce.py" rel="nofollow noreferrer">pytorch's official example</a> (See Line 71) and <a href="https://github.com/thu-ml/tianshou/blob/master/tianshou/policy/modelfree/pg.py#L226" rel="nofollow noreferrer">tianshou's example</a> (See Line 226), if I understand correctly, they are computing</p>
  <pre><code>sum_t (∇log π(a_t | s_t) * R_t)
  </code></pre>
  <p>which seems to correspond to Sutton's definition rather than Berkeley's (which will becomes</p>
  <pre><code>(sum_t ∇log π(a_t | s_t)) * (sum_t r_t)
  </code></pre>
  <p>).</p>
  <p>I wonder which definition is the de facto standard in the community. Thank you.</p>
  
Tags: <reinforcement-learning><deep-rl><reinforce>
Title: Implementation difference in REINFORCE algorithm, where to sum from
UserDisplayName: Chris XU
ThreadText: # Implementation difference in REINFORCE algorithm, where to sum from
  
  (Asked by: username_0 on 2024-01-25)
  
  I have a question regarding the implementation of the REINFORCE algorithm.
  
  In [berkeley course](https://rll.berkeley.edu/deeprlcourse/f17docs/lecture_4_policy_gradient.pdf) (see slide 9) the gradient is defined as
  [![enter image description here](https://i.sstatic.net/8uJqT.png)](https://i.sstatic.net/8uJqT.png)
  
  Note that the return **sums from 1**. However in Sutton's book the return **sums from t+1**
  [![enter image description here](https://i.sstatic.net/66LMm.png)](https://i.sstatic.net/66LMm.png)
  
  In fact in two popular implementations, [pytorch's official example](https://github.com/pytorch/examples/blob/main/reinforcement_learning/reinforce.py) (See Line 71) and [tianshou's example](https://github.com/thu-ml/tianshou/blob/master/tianshou/policy/modelfree/pg.py#L226) (See Line 226), if I understand correctly, they are computing
  
  ```
  sum_t (∇log π(a_t | s_t) * R_t)
  
  ```
  
  which seems to correspond to Sutton's definition rather than Berkeley's (which will becomes
  
  ```
  (sum_t ∇log π(a_t | s_t)) * (sum_t r_t)
  
  ```
  
  ).
  
  I wonder which definition is the de facto standard in the community. Thank you.
  
  **Tags:** reinforcement-learning, deep-rl, reinforce
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2024-01-27. Score: 2 | Accepted answer
  
  The policy gradient objective function of your Berkeley reference is *undiscounted* value function of a start state in finite horizon episodic cases, while the other Sutton's reference at least for the boxed pseudocode is about *discounted* value function of a start state in finite horizon episodic cases as stated in the book:
  
  > From here on in our discussion we will assume no discounting ($\gamma = 1$) for the episodic case, although for completeness we do include the possibility of discounting in the boxed algorithms.
  
  Since the undiscounted total return $G\_t$ is defined as $\sum\_t^T r\_t(s\_t,a\_t)$, there's no essential notation difference here. Berkeley's equation is an approximation expressed via samples $i$, so it may not be easy to use $G\_t$ without further unnecessary confusions such as $G\_{t,i}$.
  
  
  ---
  
  ### Answer by username_0 on 2024-01-26. Score: 1
  
  I think I understand now. The implementation uses a [reward to go](https://spinningup.openai.com/en/latest/spinningup/rl_intro3.html) version, which can be proved to be equivalent to the original loss, see [here](https://ai.stackexchange.com/questions/9614/why-does-the-reward-to-go-trick-in-policy-gradient-methods-work).
  
  
  ---
  
  