Answers.Body: <blockquote>
  <p>How does the neural network learn to differentiate between good and bad actions?</p>
  </blockquote>
  <p>Good actions - in context of a given state - have higher <em>return</em> than bad actions on average, taken over many examples where the actions occur in different combinations.</p>
  <p>In REINFORCE, when training the neural network, all actions are effectively treated as ground truth &quot;correct&quot;, but the gradient is weighted by the return from that time step. The best trajectories therefore get larger gradient steps, and therefore shift the action choices made in those trajectories more towards them than the less good trajectories.</p>
  <p>In the case of positive vs negative returns this is very clear (because negative returns will reverse the gradient step <em>away</em> from the action choices that were made). However, even if all returns are negative or all are positive, the REINFORCE algorithm still works - provided there are enough samples of different trajectories with actions taken in different contexts, then there will be a preference for the best action in each state.</p>
  <p>REINFORCE with baseline and Advantage-based update multipliers (which are a variation of REINFORCE with baseline) are slightly better than the most basic REINFORCE &quot;race to the top&quot;, in that they are more numerically stable and will automatically split into relative positive and negative updates regardless of whether the environment produces positive or negative returns overall.</p>
  <p>Ideally you would not run a REINFORCE update step based on a single trajectory at a time, for two reasons:</p>
  <ul>
  <li><p>The trajectory contains correlated data, and neural networks learn badly from it, preferring i.i.d. samples if possible.</p>
  </li>
  <li><p>A single trajectory does not contain enough information by itself for a learning agent to figure out whether it is good or bad relative to other options.</p>
  </li>
  </ul>
  <p>Neither of these are complete showstoppers, but you should find that many implementations of policy gradient methods collect multiple trajectories before each update step.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-25T23:10:54.427Z
Answers.Text: I'm having a hard time wrapping my head around this. Each state would have an optimal action(s) compared to other actions. How would REINFORCE learn that by viewing an entire trajectory?
  While calculating loss/backpropogation, does the algorithm ever compare the prediction for a state with the reward/value obtained?
Answers.UserDisplayName: desert_ranger
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-25T23:43:25.680Z
Answers.Text: The algorthm is gradient ascent on the neural network for the output of actions taken times the return measured when taking that action. Unlike classification which  rates the action against a known correct value, REINFORCE rates each action according to how well the overall trajectory after that point turned out. Clearly this includes impact of other choices. However, on average each state's choices will favour the best action in the long term. The algorithm never *compares* anything, it simply follows gradients that produce high returns more than it follows gradients that produce low ones.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-25T17:32:41.280Z
Answers.LastActivityDate: 2021-06-25T17:38:05.650Z
Answers.LastEditDate: 2021-06-25T17:38:05.650Z
Answers.UserDisplayName: Neil Slater
Body: <p>As per my understanding, you run an entire episode, which contains many steps, and then back-propagate using just a single loss value. How does the neural network learn to differentiate between good and bad actions?</p>
  
Tags: <neural-networks><reinforcement-learning><deep-rl><policy-gradients><reinforce>
Title: How does the neural network learn when used in the REINFORCE algorithm?
UserDisplayName: desert_ranger
ThreadText: # How does the neural network learn when used in the REINFORCE algorithm?
  
  (Asked by: username_0 on 2021-06-25)
  
  As per my understanding, you run an entire episode, which contains many steps, and then back-propagate using just a single loss value. How does the neural network learn to differentiate between good and bad actions?
  
  **Tags:** neural-networks, reinforcement-learning, deep-rl, policy-gradients, reinforce
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-06-25. Score: 3 | Accepted answer
  
  > How does the neural network learn to differentiate between good and bad actions?
  
  Good actions - in context of a given state - have higher *return* than bad actions on average, taken over many examples where the actions occur in different combinations.
  
  In REINFORCE, when training the neural network, all actions are effectively treated as ground truth "correct", but the gradient is weighted by the return from that time step. The best trajectories therefore get larger gradient steps, and therefore shift the action choices made in those trajectories more towards them than the less good trajectories.
  
  In the case of positive vs negative returns this is very clear (because negative returns will reverse the gradient step *away* from the action choices that were made). However, even if all returns are negative or all are positive, the REINFORCE algorithm still works - provided there are enough samples of different trajectories with actions taken in different contexts, then there will be a preference for the best action in each state.
  
  REINFORCE with baseline and Advantage-based update multipliers (which are a variation of REINFORCE with baseline) are slightly better than the most basic REINFORCE "race to the top", in that they are more numerically stable and will automatically split into relative positive and negative updates regardless of whether the environment produces positive or negative returns overall.
  
  Ideally you would not run a REINFORCE update step based on a single trajectory at a time, for two reasons:
  
  * The trajectory contains correlated data, and neural networks learn badly from it, preferring i.i.d. samples if possible.
  * A single trajectory does not contain enough information by itself for a learning agent to figure out whether it is good or bad relative to other options.
  
  Neither of these are complete showstoppers, but you should find that many implementations of policy gradient methods collect multiple trajectories before each update step.
  
  > **Comments:**
  > **username_0**: I'm having a hard time wrapping my head around this. Each state would have an optimal action(s) compared to other actions. How would REINFORCE learn that by viewing an entire trajectory? While calculating loss/backpropogation, does the algorithm ever compare the prediction for a state with the reward/value obtained?  
  > **username_1**: The algorthm is gradient ascent on the neural network for the output of actions taken times the return measured when taking that action. Unlike classification which  rates the action against a known correct value, REINFORCE rates each action according to how well the overall trajectory after that point turned out. Clearly this includes impact of other choices. However, on average each state's choices will favour the best action in the long term. The algorithm never *compares* anything, it simply follows gradients that produce high returns more than it follows gradients that produce low ones.  
  
  ---
  
  