Answers.Body: <blockquote>
  <p>there should be absolutely no problem with training an agent on any available episode roll-out data. That is because a MDP implies for an any state S, the optimal action to take is entirely dependent on the state.</p>
  </blockquote>
  <p>The desired end-state of the trained model is that it can identify the optimal action. When comparing reinforcement learning (RL) methods, you are looking at how they use data from experience. This data does not directly identify optimal actions, and all RL control approaches perform significant processing of the raw data in order to learn about the environment and make this identification.</p>
  <p>Any rollout may contain useful data that can be learned from for control, but it requires processing in order to drive a meaningful policy. When comparing on-policy and off-policy, you are looking at two different approaches to process this data.</p>
  <p>The differences between on-policy and off-policy methods are to do with the effectiveness of the learning algorithm in particular scenarios, and not related to using MDPs as a framework. Instead the existence of the two approaches (and other ways to solve reinforcement learning problems) are a logical <em>consequence</em> of the MDP framework.</p>
  <blockquote>
  <p>I cannot understand how old data would be harmful.</p>
  </blockquote>
  <p>When learning for optimal control, then the policy must change in order to improve. Data collected under old policies has a different distribution and results from data expected under the current policy. You can often adjust for this and still learn something (which may or may not be useful to the rest of the task) - that is exactly what off-policy learning does.</p>
  <p>Any data can be harmful in machine learning if your learning model makes incorrect assumptions about it. That is the case when an on-policy learning algorithm attempts to use data collected under a different policy. This also applies to many other types of machine learning - if something is different in an old data set to your target distribution, the trained model will be biased towards old data and make less accurate predictions on new examples.</p>
  <blockquote>
  <p>if optimal control were indeed a MDP, then there should be absolutely no problem with training an agent on any available episode roll-out data</p>
  </blockquote>
  <p>This is not true in general. Even off-policy algorithms need there to be some useful relationship to the current target policy. Data from completely random rollouts in complex problems is rarely useful for learning, except as an initial bootstrap when nothing is known about the environment.</p>
  <p>Assuming there is some useful data that could be extracted, then there is no problem training an agent, provided that agent adjusts correctly for the difference in policy. That requires off-policy agents, because on-policy agents do not have the ability to make the adjustment.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-05T19:47:07.353Z
Answers.LastActivityDate: 2022-01-06T16:12:43.270Z
Answers.LastEditDate: 2022-01-06T16:12:43.270Z
Answers.UserDisplayName: Neil Slater
Body: <p>I ask because if RL is indeed an MDP, then there should be absolutely no problem with training an agent on any available episode roll-out data, right? Because an MDP implies for any state S, the optimal action to take is entirely dependent on the state.</p>
  <p>So while I can understand the argument from <a href="https://arxiv.org/abs/1511.05952" rel="nofollow noreferrer">prioritized experience replay</a> that some data is more valuable for learning than others. I cannot understand how old data would be harmful.</p>
  <p><strong>Further Clarification:</strong></p>
  <p>The emphasis on an MDP is perhaps misleading. I don't mean to imply that optimal control should be dependent on previous states necessarily... Similarly, the emphasis on off-policy &amp; on-policy is perhaps better placed on online vs offline RL.</p>
  <p><a href="https://arxiv.org/pdf/2106.01345.pdf" rel="nofollow noreferrer">Performance Gap</a>:</p>
  <blockquote>
  <p>This [offline RL] setting is harder as it removes the ability for agents to explore the environment and collect additional feedback.</p>
  </blockquote>
  <p><strong>Consider a simple example</strong>: with Q-table 'learning'. It seems to me that Online-RL is analogous to deleting perfectly valid entries from the Q-table in order to reap better performance. This seems to me like a paradox and that's what confuses me.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-06T10:02:22.753Z
Comments.Text: I am not sure I understand this question and, in particular, I don't understand this "I ask because if optimal control were indeed a MDP" and  "performance gap between off/on-policy RL". What "gap" between on and off-policy RL are you talking about? A gap in terms of what? MDP is just a way to formulate a problem. [Optimal control](https://ai.stackexchange.com/a/23427/2444) is an area of study. Also, "I cannot understand how old data would be harmful." doesn't seem to be related to the title. Please, clarify what you're really asking here.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-06T15:07:15.170Z
Comments.Text: @nbro Neil Slater understood what I was asking so you can see his answer for context. I'm saying that off policy/offline does worse than on-policy/online, and I'm saying that I don't get why because formulating the problem as a MDP implies that any episode rollout data will always remain valid training data. So in theory excluding old data like online-RL does shouldn't increase performance yet it seems it does.
Comments.UserDisplayName: profPlum
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-06T15:53:41.310Z
Comments.Text: @profPlum: I need to understand what you mean by "performance" and what examples you have that online is "better" than offline by that measure? It surely is not by all measures - do have examples, or links to relevant statistics? Also, if my answer has not resolved your question, it would be useful to get some feedback on what you are still stuck on
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-06T15:58:19.823Z
Comments.Text: @NeilSlater I didn't have a specific metric in mind, but lets say expected accumulated reward per episode after training. Please see the citation/quote I have from Pieter Abeel stating that offline RL is 'harder'.
  
  Also you have a pretty good answer, I may accept it but just wanted to wait & see what other people say first.
Comments.UserDisplayName: profPlum
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-06T16:00:11.610Z
Comments.Text: @nbro I edited the original question to try to state what I mean more clearly. I'm sorry I probably shouldn't have placed emphasis on optimal control & MDPs. *I think the Q-table example I gave is perhaps a better way of stating the question.*
Comments.UserDisplayName: profPlum
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-06T16:57:25.477Z
Comments.Text: @NeilSlater Actually probably the only thing I'm still stuck on is the Q-table example I gave. In such an admittedly trivial/hypothetical situation, it is so obvious that extra (correct) data shouldn't be harmful. Is the difference here specifically *because we are using a NN* to approximate the table?
Comments.UserDisplayName: profPlum
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-06T17:12:20.723Z
Comments.Text: @profPlum There are many ways to store and update using historical data from trajectories. You would need to be more precise about the algorithm you intend to use before I could answer you about potential harm. For instance, a simple implementation of single-step SARSA would have problems because the next action $a'$ is required, and may be incorrect in the historical data, causing the TD target to be incorrect. You could resample $a'$ or used Expected SARSA, but that changes your algorithm from on-policy to off-policy.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-06T21:21:17.110Z
Comments.Text: @profPlum Now that you think you know what you want to ask, can you please remove all the unnecessary parts of this post and just leave what you think is clear and your actual question? Right now, I still don't understand what's really being asked and I also don't understand your example. How is "Online-RL is analogous to deleting perfectly valid entries from the Q-table in order to reap better performance" true? It makes no sense to me this statement. Do you know the difference between online and on-policy? If not, read [this](https://ai.stackexchange.com/q/10474/2444) before proceeding.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-06T21:27:19.163Z
Comments.Text: From the title, it seems that you're asking "Why can't we use old data for on-policy algorithms?". If that is your question, I guess you're trying to understand the difference between on-policy and off-policy. If that's not your question, then I have no idea of what you're asking.
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><deep-rl>
Title: Why is old/off-policy data harmful to on-policy/online RL?
UserDisplayName: profPlum
ThreadText: # Why is old/off-policy data harmful to on-policy/online RL?
  
  (Asked by: username_0 on 2022-01-05)
  
  I ask because if RL is indeed an MDP, then there should be absolutely no problem with training an agent on any available episode roll-out data, right? Because an MDP implies for any state S, the optimal action to take is entirely dependent on the state.
  
  So while I can understand the argument from [prioritized experience replay](https://arxiv.org/abs/1511.05952) that some data is more valuable for learning than others. I cannot understand how old data would be harmful.
  
  **Further Clarification:**
  
  The emphasis on an MDP is perhaps misleading. I don't mean to imply that optimal control should be dependent on previous states necessarily... Similarly, the emphasis on off-policy & on-policy is perhaps better placed on online vs offline RL.
  
  [Performance Gap](https://arxiv.org/pdf/2106.01345.pdf):
  
  > This [offline RL] setting is harder as it removes the ability for agents to explore the environment and collect additional feedback.
  
  **Consider a simple example**: with Q-table 'learning'. It seems to me that Online-RL is analogous to deleting perfectly valid entries from the Q-table in order to reap better performance. This seems to me like a paradox and that's what confuses me.
  
  **Tags:** reinforcement-learning, deep-rl
  > **Comments:**
  > **username_1**: I am not sure I understand this question and, in particular, I don't understand this "I ask because if optimal control were indeed a MDP" and  "performance gap between off/on-policy RL". What "gap" between on and off-policy RL are you talking about? A gap in terms of what? MDP is just a way to formulate a problem. [Optimal control](https://ai.stackexchange.com/a/23427/2444) is an area of study. Also, "I cannot understand how old data would be harmful." doesn't seem to be related to the title. Please, clarify what you're really asking here.  
  > **username_0**: @nbro Neil Slater understood what I was asking so you can see his answer for context. I'm saying that off policy/offline does worse than on-policy/online, and I'm saying that I don't get why because formulating the problem as a MDP implies that any episode rollout data will always remain valid training data. So in theory excluding old data like online-RL does shouldn't increase performance yet it seems it does.  
  > **username_2**: @profPlum: I need to understand what you mean by "performance" and what examples you have that online is "better" than offline by that measure? It surely is not by all measures - do have examples, or links to relevant statistics? Also, if my answer has not resolved your question, it would be useful to get some feedback on what you are still stuck on  
  > **username_0**: @NeilSlater I didn't have a specific metric in mind, but lets say expected accumulated reward per episode after training. Please see the citation/quote I have from Pieter Abeel stating that offline RL is 'harder'.  Also you have a pretty good answer, I may accept it but just wanted to wait & see what other people say first.  
  > **username_0**: @nbro I edited the original question to try to state what I mean more clearly. I'm sorry I probably shouldn't have placed emphasis on optimal control & MDPs. *I think the Q-table example I gave is perhaps a better way of stating the question.*  
  > **username_0**: @NeilSlater Actually probably the only thing I'm still stuck on is the Q-table example I gave. In such an admittedly trivial/hypothetical situation, it is so obvious that extra (correct) data shouldn't be harmful. Is the difference here specifically *because we are using a NN* to approximate the table?  
  > **username_2**: @profPlum There are many ways to store and update using historical data from trajectories. You would need to be more precise about the algorithm you intend to use before I could answer you about potential harm. For instance, a simple implementation of single-step SARSA would have problems because the next action $a'$ is required, and may be incorrect in the historical data, causing the TD target to be incorrect. You could resample $a'$ or used Expected SARSA, but that changes your algorithm from on-policy to off-policy.  
  > **username_1**: @profPlum Now that you think you know what you want to ask, can you please remove all the unnecessary parts of this post and just leave what you think is clear and your actual question? Right now, I still don't understand what's really being asked and I also don't understand your example. How is "Online-RL is analogous to deleting perfectly valid entries from the Q-table in order to reap better performance" true? It makes no sense to me this statement. Do you know the difference between online and on-policy? If not, read [this](https://ai.stackexchange.com/q/10474/2444) before proceeding.  
  > **username_1**: From the title, it seems that you're asking "Why can't we use old data for on-policy algorithms?". If that is your question, I guess you're trying to understand the difference between on-policy and off-policy. If that's not your question, then I have no idea of what you're asking.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-01-05. Score: 1
  
  > there should be absolutely no problem with training an agent on any available episode roll-out data. That is because a MDP implies for an any state S, the optimal action to take is entirely dependent on the state.
  
  The desired end-state of the trained model is that it can identify the optimal action. When comparing reinforcement learning (RL) methods, you are looking at how they use data from experience. This data does not directly identify optimal actions, and all RL control approaches perform significant processing of the raw data in order to learn about the environment and make this identification.
  
  Any rollout may contain useful data that can be learned from for control, but it requires processing in order to drive a meaningful policy. When comparing on-policy and off-policy, you are looking at two different approaches to process this data.
  
  The differences between on-policy and off-policy methods are to do with the effectiveness of the learning algorithm in particular scenarios, and not related to using MDPs as a framework. Instead the existence of the two approaches (and other ways to solve reinforcement learning problems) are a logical *consequence* of the MDP framework.
  
  > I cannot understand how old data would be harmful.
  
  When learning for optimal control, then the policy must change in order to improve. Data collected under old policies has a different distribution and results from data expected under the current policy. You can often adjust for this and still learn something (which may or may not be useful to the rest of the task) - that is exactly what off-policy learning does.
  
  Any data can be harmful in machine learning if your learning model makes incorrect assumptions about it. That is the case when an on-policy learning algorithm attempts to use data collected under a different policy. This also applies to many other types of machine learning - if something is different in an old data set to your target distribution, the trained model will be biased towards old data and make less accurate predictions on new examples.
  
  > if optimal control were indeed a MDP, then there should be absolutely no problem with training an agent on any available episode roll-out data
  
  This is not true in general. Even off-policy algorithms need there to be some useful relationship to the current target policy. Data from completely random rollouts in complex problems is rarely useful for learning, except as an initial bootstrap when nothing is known about the environment.
  
  Assuming there is some useful data that could be extracted, then there is no problem training an agent, provided that agent adjusts correctly for the difference in policy. That requires off-policy agents, because on-policy agents do not have the ability to make the adjustment.
  
  
  ---
  
  