Answers.Body: <p>Using a &quot;memory&quot; of previous experience with the REINFORCE algorithm will not work. The algorithm relies heavily on the training data distribution matching current on-policy behaviour.</p>
  <p>Learning the optimal policy in REINFORCE can be framed as a race where whatever the current policy choice is, is treated as correct, weighted only by return value and distribution. Using a history for training will either select that whole history as correct, or reject it (constructing some kind of probably non optimal anti-policy) depending on whether the returns at that time were positive or negative. Those returns will most likely be incorrect as they will be due to combined choices made when the policy was more random. And even if they are correct, if there is selection bias in the distribution, it will very heavily skewed results.</p>
  <p>For example, if in state <span class="math-container">$s$</span>, the history contains two examples of selecting action <span class="math-container">$a_1$</span> with return 2 and one of <span class="math-container">$a_2$</span> with return 3, then REINFORCE repeatedly trained on this will prefer action <span class="math-container">$a_1$</span>. This effect can still happen on-policy, but is mitigated by the change in policy that occurs dynamically during on-policy sampling.</p>
  <p>There are ways to use policy gradient in off-policy. One is the method Deep Deterministic Policy Gradient (DDPG), where the policy network outputs a single action choice, and a separate noise function is added to that choice to collect experience.</p>
  <p>Also actor-critic methods can use off-policy approaches to train the critic, and could use a replay memory for that</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-17T12:00:34.817Z
Answers.Text: I will need to check whether DDPG is used with a history
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-17T13:13:38.637Z
Answers.Text: Yes, DDPG is off-policy and uses a replay buffer. @NeilSlater
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-26T09:27:08.500Z
Answers.Text: @NeilSlater For those that might be implementing REINFORCE with baseline though, the second Neural Network that learns the state-value function can implement a memory though, right? I understand the policy network can't thanks to your example but I think it would be ok to have a memory for the state-value NN? Although I'm not sure since if a bad decision is over-represented in the memory it might be problematic? But that argument could be made for Q-learning and we do use a memory in Q-learning.
Answers.UserDisplayName: FluidMechanics Potential Flows
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-26T21:17:38.180Z
Answers.Text: @FluidMechanicsPotentialFlows Yes that would be similar to the actor-critic approach as long as you use an off-policy learning algorithm for the state value function (assuming you are using state value for the baseline).
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-27T20:01:02.687Z
Answers.Text: What would an off-policy learning algorithm for the state value function be? I was thinking to use a Neural Network that tries to minimise its prediction of the reward at a given state with the reward actually received at this given state, following the REINFORCE policy. Does this have a name? It seems similar to Q-learning but it's not really learning Q-values.
Answers.UserDisplayName: FluidMechanics Potential Flows
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-27T21:31:26.307Z
Answers.Text: @FluidMechanicsPotentialFlows I think that might make a good separate question. I'm not 100% certain, but think that even for TD learning single step - aka TD(0) - in this case you will need to use importance sampling adjustment which will amount to ignoring exploratory action choices if you want to baseline on the best guess optimal greedy action. It might be a lot of effort for very little practical effect (you may get nicer value estimates for the baseline, but the impact of that is kind of secondary given that even a single fixed baseline for all states is still somewhat useful)
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-28T06:58:40.160Z
Answers.Text: I see, what do you mean by a single fixed baseline for all states though? I'm not sure I follow that part. In a symmetrical zero-sum game typically, I'm not sure I can fix a baseline for all states that would be different than 0. Maybe you were mentioning environments where there is not "this symmetry" around 0?
Answers.UserDisplayName: FluidMechanics Potential Flows
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-28T07:36:57.457Z
Answers.Text: @FluidMechanicsPotentialFlows even in a zero sum game, the mean reward for current or optimal policies may not be zero. And there are plenty of environments that are not zero sum games balanced evenly between players. It's entirely possible for some environments that adding a baseline to REINFORCE method makes only a minor improvement to variance
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-28T11:04:00.197Z
Answers.Text: I see, indeed, thank you for the insights
Answers.UserDisplayName: FluidMechanics Potential Flows
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-13T18:02:29.197Z
Answers.Text: @NeilSlater I'm coming back to your answer after giving it some further thoughts. If you implement REINFORCE with a single fixed baseline and you use a memory, you might see reward 2 twice as often but eventually your baseline will get close to 2 / catch up and only the reward 3 will change the probabilities? And after the baseline gets over 2 the loss will be such that the network will actually want to avoid what led to reward 2?
Answers.UserDisplayName: FluidMechanics Potential Flows
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-13T20:13:01.000Z
Answers.Text: @FluidMechanicsPotentialFlows Yes, a baseline approach would help solve the specific 2/3 issue here if they were the only data points because the 2s would become negative with the subtracted basline (of 2.33) and the 3 would still be positive. However, you could probably still come up with some combination of values where REINFORCE with baseline is fooled into preferring lower returns with higher frequencies, it just becomes less easy to get into that situation.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-14T09:16:52.677Z
Answers.Text: If it prefers lower returns with higher frequencies it still means that the baseline is below those returns right? So it would "improve" the model if we are just looking at the baseline/average reward? I guess the point you make is that the model could be improved better if we were to discard them?
Answers.UserDisplayName: FluidMechanics Potential Flows
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-14T14:12:13.813Z
Answers.Text: No it doesn't. You could have returns 0, 0, 0, 1, 1, 1, 2, 2, 2, 3. The baseline using mean return is 1.2, and frequency times baseline returns would be 0: -1.2, 1 -0.6, 2: +2.4, 3: +1.8, so the algorithm will promote the return of 2.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-08-02T10:53:43.857Z
Answers.Text: @NeilSlater yes but after it gets to 2 on average, the baseline will get to 2 and it will now promote the return of 3?
Answers.UserDisplayName: FluidMechanics Potential Flows
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-08-02T12:05:00.243Z
Answers.Text: @FluidMechanicsPotentialFlows we are considering the impact of a current history only. Yes there are dynamics that might help, but there's also a race condition here where the policy will have locked in to the action with return of 2 harder. These constructed examples are mostly showing you inefficiencies, not hard failures though. You might "get away with it", but you will be better off not trying to use a replay memory with a strictly online learning method
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-17T11:59:57.723Z
Answers.LastActivityDate: 2024-05-17T12:11:13.017Z
Answers.LastEditDate: 2024-05-17T12:11:13.017Z
Answers.UserDisplayName: Neil Slater
Body: <p>In Q-learning with function approximators such as Neural Networks, we typically implement a memory so that at the end of each episode we also train on past experiences. This is typically fine because Q-learning is off-policy learning. Is it the case that this would not work with the REINFORCE algorithm? Is there a &quot;mathematical&quot; reason why this would be the case? I can't figure if &quot;off-policy&quot; learning would be detrimental to REINFORCE?</p>
  
Tags: <reinforcement-learning><deep-rl><reinforce>
Title: Can we implement a memory in a REINFORCE algorithm for RL?
UserDisplayName: FluidMechanics Potential Flows
ThreadText: # Can we implement a memory in a REINFORCE algorithm for RL?
  
  (Asked by: username_0 on 2024-05-17)
  
  In Q-learning with function approximators such as Neural Networks, we typically implement a memory so that at the end of each episode we also train on past experiences. This is typically fine because Q-learning is off-policy learning. Is it the case that this would not work with the REINFORCE algorithm? Is there a "mathematical" reason why this would be the case? I can't figure if "off-policy" learning would be detrimental to REINFORCE?
  
  **Tags:** reinforcement-learning, deep-rl, reinforce
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-05-17. Score: 5 | Accepted answer
  
  Using a "memory" of previous experience with the REINFORCE algorithm will not work. The algorithm relies heavily on the training data distribution matching current on-policy behaviour.
  
  Learning the optimal policy in REINFORCE can be framed as a race where whatever the current policy choice is, is treated as correct, weighted only by return value and distribution. Using a history for training will either select that whole history as correct, or reject it (constructing some kind of probably non optimal anti-policy) depending on whether the returns at that time were positive or negative. Those returns will most likely be incorrect as they will be due to combined choices made when the policy was more random. And even if they are correct, if there is selection bias in the distribution, it will very heavily skewed results.
  
  For example, if in state $s$, the history contains two examples of selecting action $a\_1$ with return 2 and one of $a\_2$ with return 3, then REINFORCE repeatedly trained on this will prefer action $a\_1$. This effect can still happen on-policy, but is mitigated by the change in policy that occurs dynamically during on-policy sampling.
  
  There are ways to use policy gradient in off-policy. One is the method Deep Deterministic Policy Gradient (DDPG), where the policy network outputs a single action choice, and a separate noise function is added to that choice to collect experience.
  
  Also actor-critic methods can use off-policy approaches to train the critic, and could use a replay memory for that
  
  > **Comments:**
  > **username_1**: I will need to check whether DDPG is used with a history  
  > **username_2**: Yes, DDPG is off-policy and uses a replay buffer. @NeilSlater  
  > **username_0**: @NeilSlater For those that might be implementing REINFORCE with baseline though, the second Neural Network that learns the state-value function can implement a memory though, right? I understand the policy network can't thanks to your example but I think it would be ok to have a memory for the state-value NN? Although I'm not sure since if a bad decision is over-represented in the memory it might be problematic? But that argument could be made for Q-learning and we do use a memory in Q-learning.  
  > **username_1**: @FluidMechanicsPotentialFlows Yes that would be similar to the actor-critic approach as long as you use an off-policy learning algorithm for the state value function (assuming you are using state value for the baseline).  
  > **username_0**: What would an off-policy learning algorithm for the state value function be? I was thinking to use a Neural Network that tries to minimise its prediction of the reward at a given state with the reward actually received at this given state, following the REINFORCE policy. Does this have a name? It seems similar to Q-learning but it's not really learning Q-values.  
  > **username_1**: @FluidMechanicsPotentialFlows I think that might make a good separate question. I'm not 100% certain, but think that even for TD learning single step - aka TD(0) - in this case you will need to use importance sampling adjustment which will amount to ignoring exploratory action choices if you want to baseline on the best guess optimal greedy action. It might be a lot of effort for very little practical effect (you may get nicer value estimates for the baseline, but the impact of that is kind of secondary given that even a single fixed baseline for all states is still somewhat useful)  
  > **username_0**: I see, what do you mean by a single fixed baseline for all states though? I'm not sure I follow that part. In a symmetrical zero-sum game typically, I'm not sure I can fix a baseline for all states that would be different than 0. Maybe you were mentioning environments where there is not "this symmetry" around 0?  
  > **username_1**: @FluidMechanicsPotentialFlows even in a zero sum game, the mean reward for current or optimal policies may not be zero. And there are plenty of environments that are not zero sum games balanced evenly between players. It's entirely possible for some environments that adding a baseline to REINFORCE method makes only a minor improvement to variance  
  > **username_0**: I see, indeed, thank you for the insights  
  > **username_0**: @NeilSlater I'm coming back to your answer after giving it some further thoughts. If you implement REINFORCE with a single fixed baseline and you use a memory, you might see reward 2 twice as often but eventually your baseline will get close to 2 / catch up and only the reward 3 will change the probabilities? And after the baseline gets over 2 the loss will be such that the network will actually want to avoid what led to reward 2?  
  > **username_1**: @FluidMechanicsPotentialFlows Yes, a baseline approach would help solve the specific 2/3 issue here if they were the only data points because the 2s would become negative with the subtracted basline (of 2.33) and the 3 would still be positive. However, you could probably still come up with some combination of values where REINFORCE with baseline is fooled into preferring lower returns with higher frequencies, it just becomes less easy to get into that situation.  
  > **username_0**: If it prefers lower returns with higher frequencies it still means that the baseline is below those returns right? So it would "improve" the model if we are just looking at the baseline/average reward? I guess the point you make is that the model could be improved better if we were to discard them?  
  > **username_1**: No it doesn't. You could have returns 0, 0, 0, 1, 1, 1, 2, 2, 2, 3. The baseline using mean return is 1.2, and frequency times baseline returns would be 0: -1.2, 1 -0.6, 2: +2.4, 3: +1.8, so the algorithm will promote the return of 2.  
  > **username_0**: @NeilSlater yes but after it gets to 2 on average, the baseline will get to 2 and it will now promote the return of 3?  
  > **username_1**: @FluidMechanicsPotentialFlows we are considering the impact of a current history only. Yes there are dynamics that might help, but there's also a race condition here where the policy will have locked in to the action with return of 2 harder. These constructed examples are mostly showing you inefficiencies, not hard failures though. You might "get away with it", but you will be better off not trying to use a replay memory with a strictly online learning method  
  
  ---
  
  