Answers.Body: <p>One simple improvement over the REINFORCE algorithm you've linked to is to use the advantage function instead of the normalised cumulative discounted return. The implementation is can be found in the same folder in <a href="https://github.com/pytorch/examples/blob/main/reinforcement_learning/actor_critic.py" rel="nofollow noreferrer">actor_critic.py</a>.</p>
  <p>In your implementation of REINFORCE, the gradient of the loss is calculated as:
  <span class="math-container">$$
  L(\theta) = -\sum_{t=0}^T\hat{G}_t\log(\pi_\theta(a_t|s_t))
  $$</span>
  with
  <span class="math-container">$$\hat{G}_t = \frac{\sum_{k=0}^t r_k - \hat{r}}{\hat{\sigma_r} + \epsilon}$$</span>
  where <span class="math-container">$\hat{r}$</span> is the empirical mean over the trajectory, <span class="math-container">$\hat{\sigma_r}$</span> the empirical standard deviation, and <span class="math-container">$\epsilon$</span> a constant to ensure the denominator is not too close to zero.</p>
  <p>Now, actor_critic.py replaces <span class="math-container">$\hat{G}_t$</span> in the loss with <span class="math-container">$\hat{G}_t - V_\phi(s_t)$</span>, called the advantage function, where <span class="math-container">$V_\phi$</span> is the state value function, which is a neural network with parameters <span class="math-container">$\phi$</span>. <span class="math-container">$V_\phi$</span> is learnt, so the actual loss you optimize is:
  <span class="math-container">$$J(\theta,\phi) = L(\theta) + \sum_{t=0}^T\left(V_\phi(s_t) - \hat{G}_t\right)^2$$</span></p>
  <p>This method is superior to your plain REINFORCE because the variance of the gradient updates is lower (which means learning is more steady and you obtain a better estimate of the optimal policy <span class="math-container">$\pi^*$</span>).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-09-13T00:02:40.877Z
Answers.Text: Am I correct in understanding that, even though the actor and the critic are the same network except the last layer, that change is sufficient enough for the network to provide the state value? And event though the policy is providing the action at step t (which creates a new state t+1), the critic is providing the value for the new state (t+1)?
Answers.UserDisplayName: Josh
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-09-14T19:35:12.167Z
Answers.Text: Yes, it's common for the value and the policy network to share most of their weights, except the final layer(s). You can play around with what's shared and what's not though, there are no rules. I'm not sure why you say that the critic is providing the value for the new state: the idea of the advantage is to tell you how much better the return you get by taking action $a_t$ is compared to the return of the "average" action (what the value function tells you)
Answers.UserDisplayName: Raphael Lopez Kaufman
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-09-12T21:51:20.853Z
Answers.LastActivityDate: 2022-09-14T19:35:29.753Z
Answers.LastEditDate: 2022-09-14T19:35:29.753Z
Answers.UserDisplayName: Raphael Lopez Kaufman
Body: <p>I am straying out of my domain knowledge to attempt a basic reinforcement learning task in a toy environment and have become fairly familiar with the REINFORCE algorithm for policy gradient agents, especially PyTorch's implementation (<a href="https://github.com/pytorch/examples/blob/main/reinforcement_learning/reinforce.py" rel="nofollow noreferrer">found here</a>). It is clear to me now that there are superior methods to train RL agents (PPO for instance), but as I read, these feel beyond my current intellectual or time resources. As such, I'd like to eek out as much power through modifications of REINFORCE as possible before determining how I might move on.</p>
  <p>As such, are there modifications to the REINFORCE training algorithm that might yield benefits without straying far into new algorithm territory? Or perhaps, what is the &quot;SOTA&quot; version of REINFORCE?</p>
  <p>For instance, perhaps a simple gradient clip in some way approximates some of PPO's benefits? Or maybe setting a baseline reward based on a rolling reward set of previous episodes?</p>
  <hr />
  <p>If a specific context is useful, I'm applying this to a self-made simple grid environment where agents receive points namely for moving closer to and acquiring &quot;targets.&quot; There are other rewards of lesser significance, but key is the environment is set for a sort of &quot;continuous play&quot; such that agents are very frequently receiving rewards (due to closeness) and occasionally receiving reward spikes (due to getting a target), but there is no true episode definition other than an arbitrary timestep length. I am not using batches (perhaps that is useful?), as I have found that gradually stepping up the episode length allows the agents quicker access to simpler rewards and appears a sort of scaffolding to more complex behavior. Agents might reasonably gather the &quot;large&quot; rewards in as few as 10-25 steps. Generally things are working fine, and I am most interested in how to extract as much value from the agent updating mechanism (REINFORCE) as possible.</p>
  
Tags: <reinforcement-learning><deep-rl><policy-gradients><reinforce>
Title: What modifications can maximize the efficacy of the REINFORCE algorithm for a policy gradient task?
UserDisplayName: Josh
ThreadText: # What modifications can maximize the efficacy of the REINFORCE algorithm for a policy gradient task?
  
  (Asked by: username_0 on 2022-09-10)
  
  I am straying out of my domain knowledge to attempt a basic reinforcement learning task in a toy environment and have become fairly familiar with the REINFORCE algorithm for policy gradient agents, especially PyTorch's implementation ([found here](https://github.com/pytorch/examples/blob/main/reinforcement_learning/reinforce.py)). It is clear to me now that there are superior methods to train RL agents (PPO for instance), but as I read, these feel beyond my current intellectual or time resources. As such, I'd like to eek out as much power through modifications of REINFORCE as possible before determining how I might move on.
  
  As such, are there modifications to the REINFORCE training algorithm that might yield benefits without straying far into new algorithm territory? Or perhaps, what is the "SOTA" version of REINFORCE?
  
  For instance, perhaps a simple gradient clip in some way approximates some of PPO's benefits? Or maybe setting a baseline reward based on a rolling reward set of previous episodes?
  
  ---
  
  If a specific context is useful, I'm applying this to a self-made simple grid environment where agents receive points namely for moving closer to and acquiring "targets." There are other rewards of lesser significance, but key is the environment is set for a sort of "continuous play" such that agents are very frequently receiving rewards (due to closeness) and occasionally receiving reward spikes (due to getting a target), but there is no true episode definition other than an arbitrary timestep length. I am not using batches (perhaps that is useful?), as I have found that gradually stepping up the episode length allows the agents quicker access to simpler rewards and appears a sort of scaffolding to more complex behavior. Agents might reasonably gather the "large" rewards in as few as 10-25 steps. Generally things are working fine, and I am most interested in how to extract as much value from the agent updating mechanism (REINFORCE) as possible.
  
  **Tags:** reinforcement-learning, deep-rl, policy-gradients, reinforce
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-09-12. Score: 4 | Accepted answer
  
  One simple improvement over the REINFORCE algorithm you've linked to is to use the advantage function instead of the normalised cumulative discounted return. The implementation is can be found in the same folder in [actor\_critic.py](https://github.com/pytorch/examples/blob/main/reinforcement_learning/actor_critic.py).
  
  In your implementation of REINFORCE, the gradient of the loss is calculated as:
  $$
  L(\theta) = -\sum\_{t=0}^T\hat{G}\_t\log(\pi\_\theta(a\_t|s\_t))
  $$
  with
  $$\hat{G}\_t = \frac{\sum\_{k=0}^t r\_k - \hat{r}}{\hat{\sigma\_r} + \epsilon}$$
  where $\hat{r}$ is the empirical mean over the trajectory, $\hat{\sigma\_r}$ the empirical standard deviation, and $\epsilon$ a constant to ensure the denominator is not too close to zero.
  
  Now, actor\_critic.py replaces $\hat{G}\_t$ in the loss with $\hat{G}\_t - V\_\phi(s\_t)$, called the advantage function, where $V\_\phi$ is the state value function, which is a neural network with parameters $\phi$. $V\_\phi$ is learnt, so the actual loss you optimize is:
  $$J(\theta,\phi) = L(\theta) + \sum\_{t=0}^T\left(V\_\phi(s\_t) - \hat{G}\_t\right)^2$$
  
  This method is superior to your plain REINFORCE because the variance of the gradient updates is lower (which means learning is more steady and you obtain a better estimate of the optimal policy $\pi^\*$).
  
  > **Comments:**
  > **username_0**: Am I correct in understanding that, even though the actor and the critic are the same network except the last layer, that change is sufficient enough for the network to provide the state value? And event though the policy is providing the action at step t (which creates a new state t+1), the critic is providing the value for the new state (t+1)?  
  > **username_1**: Yes, it's common for the value and the policy network to share most of their weights, except the final layer(s). You can play around with what's shared and what's not though, there are no rules. I'm not sure why you say that the critic is providing the value for the new state: the idea of the advantage is to tell you how much better the return you get by taking action $a_t$ is compared to the return of the "average" action (what the value function tells you)  
  
  ---
  
  