Answers.Body: <p>The main similarity between reinforcement learning experience and supervised learning datasets, is that both consist of a set of records. These records are commonly expressed as vectors of numbers for use in the algorithms. In addition, reinforcement learning that uses neural networks (or other function approximation) will typically implement some variant of supervised learning internally.</p>
  <p>There are a few key differences between a prepared dataset for supervised learning and the experience in reinforcement learning. There are exceptions to these, but these are the usual case:</p>
  <ul>
  <li><p>A supervised learning dataset has a fixed target value to learn by association, for each entry, e.g. a class or regression value. An individual reinforcement learning experience does not - the raw tuple of state, action, reward, next state <span class="math-container">$(s,a,r,s')$</span> must be processed in some way to obtain a useful training target value, and this processing is <em>not static</em>. When training in reinforcement learning for optimal control, even with historical experience, the target values must constantly be re-assessed.</p>
  </li>
  <li><p>Reinforcement learning is in part a design for actively collecting experience. There is no equivalent in supervised learning where the dataset is a given.</p>
  </li>
  <li><p>Reinforcement learning experience arrives in groups of <span class="math-container">$(s, a, r, s')$</span> - state, action, reward, next state - such groups of related data within a record are called <em>tuples</em>. The RL records are often correlated with each other, at least initially because each time step changes things only slightly, and that can be bad when combined with supervised learning which usually assumes uncorrelated data. In supervised learning you will often have a deliberate shuffling or randomisation of dataset order to protect against this. Experience replay in deep RL is a related idea to protect internal neural networks from being exposed to training samples in sequences with correlated values.</p>
  </li>
  </ul>
  <p>It is possible to apply reinforcement learning to supervised learning problems in theory. You can do this by making the agent guess each labelled value as an action, and reward it with negative the loss from the supervised learning. This is generally a bad idea because it is very inefficient, and there is no matching concept of state transitions in the supervised learning problem (the agent cannot impact what state is next due to its guess). However, the fact that this is possible with very little modification to the reinforcement learning agent shows how general reinforcement learning is as a learning approach.</p>
  <p>The inverse is not really true, you cannot normally adjust a supervised learning algorithm so that it can solve a reinforcement learning problem from the given experience. There are some edge cases, such as when learning only from previous experience to assess an existing policy, or to learn a control algorithm which only cares about immediate reward. In which case you could use reinforcement learning theory to help construct a fixed dataset and give that to a supervised learning algorithm. However, by far the most common approach is to use supervised learning approaches (e.g. a neural network) as components of the agent, and rely on reinforcement learning to generate data for them on the fly.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-06T07:52:01.657Z
Answers.LastActivityDate: 2021-09-07T17:49:45.110Z
Answers.LastEditDate: 2021-09-07T17:49:45.110Z
Answers.UserDisplayName: Neil Slater
Body: <p>Statistics is a branch of mathematics that extracts useful information from data. The data is generally called as &quot;training data&quot; in statistical (machine) learning.</p>
  <p>Consider the following paragraph from the <code>section 1.1 Reinforcement Learning</code> of <code>CHAPTER 1. THE REINFORCEMENT LEARNING PROBLEM</code> of the textbook <a href="https://web.stanford.edu/class/psych209/Readings/SuttonBartoIPRLBook2ndEd.pdf" rel="nofollow noreferrer">Reinforcement Learning: An Introduction</a> by Richard S. Sutton and Andrew G. Barto.</p>
  <blockquote>
  <p>Reinforcement learning
  is different from supervised learning, the kind of learning studied in
  most current research in field of machine learning. Supervised
  learning is learning from a training set of labeled examples provided
  by a knowledgable external supervisor. Each example is a description
  of a situation together with a specification—the label—of the correct
  action the system should take to that situation, which is often to
  identify a category to which the situation belongs. The object of this
  kind of learning is for the system to extrapolate, or generalize, its
  responses so that it acts correctly in situations <strong>not present in the
  training set</strong>. This is an important kind of learning, <strong>but alone it is
  not adequate for learning from interaction</strong>. In interactive problems it
  is often <strong>impractical</strong> to obtain examples of desired behavior that are
  both correct and representative of all the situations in which the
  agent has to act. In uncharted territory—where one would expect
  learning to be most beneficial—an agent must be able to learn from its
  own experience.</p>
  </blockquote>
  <p>You can observe that, training data in machine learning, if we model it in a proper format, can be for reinforcement learning. But, may not be complete and practical.</p>
  <p>I am asking this question in the view of statistical learning rather than machine learning alone. training data in statistical learning can be understood to any data, at any time instant, useful in learning.</p>
  <p>Then, is it perfectly fine to always interpret experience in reinforcement learning as training data in statistical learning?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-05T11:30:42.770Z
Comments.Text: When you say "I am asking this question in the view of statistical learning rather than machine learning alone.", you need to clarify what you mean by both, because, from my perspective, statistical learning is just another name for ML, although you may put emphasis on statistics, or, put in another way, to me it's just ML with a statistical perspective, but given that ML and statistics overlap a lot, then the difference is really minimal if none and this is just a problem of consensus and/or terminology.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-05T11:33:10.907Z
Comments.Text: Having said that, I am not sure I understand the type of answer you're looking for. My answer would be "Yes, experience would be some type of data. We call it "experience" because we can collect more and more, as opposed to SL, where you typically have a fixed dataset, while in RL, at least in simulation, this is not the case, although with things like experience replay, one may think of it as collecting a dataset, but this dataset is not fixed and can actually change anyway." I will not post this as a formal answer until you clarify the type of answer you're looking for.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-05T12:01:34.740Z
Comments.Text: @nbro Although they are same, I am guessing that there may be some differences if we go too deep in to formal aspects. So. to keep them intact at all levels. I used two different names.
Comments.UserDisplayName: hanugm
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-05T12:02:39.633Z
Comments.Text: If I always start interpreting experience in RL as the data in statistical AI, then will I get contradiction somewhere or not is the question.
Comments.UserDisplayName: hanugm
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-05T13:06:54.113Z
Comments.Text: I do not understand this question, I think because you do not clarify anywahere what you mean by "treat like" from the title. Or "interpret" in the question body. Assuming there are some useful parallels between the two terms (and I think there are), what do you intend to **do** with that information? Are you looking for ways to transform between supervised learning and RL, or trying to understand the differences between RL experience and a supervised learning dataset?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-05T21:38:27.403Z
Comments.Text: @NeilSlater trying to understand difference between RL experience and SL data.
Comments.UserDisplayName: hanugm
Tags: <reinforcement-learning><comparison><statistical-ai>
Title: Can I treat "experience" in reinforcement learning as "training data" in statistical learning?
UserDisplayName: hanugm
ThreadText: # Can I treat "experience" in reinforcement learning as "training data" in statistical learning?
  
  (Asked by: username_0 on 2021-09-05)
  
  Statistics is a branch of mathematics that extracts useful information from data. The data is generally called as "training data" in statistical (machine) learning.
  
  Consider the following paragraph from the `section 1.1 Reinforcement Learning` of `CHAPTER 1. THE REINFORCEMENT LEARNING PROBLEM` of the textbook [Reinforcement Learning: An Introduction](https://web.stanford.edu/class/psych209/Readings/SuttonBartoIPRLBook2ndEd.pdf) by Richard S. Sutton and Andrew G. Barto.
  
  > Reinforcement learning
  > is different from supervised learning, the kind of learning studied in
  > most current research in field of machine learning. Supervised
  > learning is learning from a training set of labeled examples provided
  > by a knowledgable external supervisor. Each example is a description
  > of a situation together with a specification—the label—of the correct
  > action the system should take to that situation, which is often to
  > identify a category to which the situation belongs. The object of this
  > kind of learning is for the system to extrapolate, or generalize, its
  > responses so that it acts correctly in situations **not present in the
  > training set**. This is an important kind of learning, **but alone it is
  > not adequate for learning from interaction**. In interactive problems it
  > is often **impractical** to obtain examples of desired behavior that are
  > both correct and representative of all the situations in which the
  > agent has to act. In uncharted territory—where one would expect
  > learning to be most beneficial—an agent must be able to learn from its
  > own experience.
  
  You can observe that, training data in machine learning, if we model it in a proper format, can be for reinforcement learning. But, may not be complete and practical.
  
  I am asking this question in the view of statistical learning rather than machine learning alone. training data in statistical learning can be understood to any data, at any time instant, useful in learning.
  
  Then, is it perfectly fine to always interpret experience in reinforcement learning as training data in statistical learning?
  
  **Tags:** reinforcement-learning, comparison, statistical-ai
  
  > **Comments:**
  > **username_1**: When you say "I am asking this question in the view of statistical learning rather than machine learning alone.", you need to clarify what you mean by both, because, from my perspective, statistical learning is just another name for ML, although you may put emphasis on statistics, or, put in another way, to me it's just ML with a statistical perspective, but given that ML and statistics overlap a lot, then the difference is really minimal if none and this is just a problem of consensus and/or terminology.  
  > **username_1**: Having said that, I am not sure I understand the type of answer you're looking for. My answer would be "Yes, experience would be some type of data. We call it "experience" because we can collect more and more, as opposed to SL, where you typically have a fixed dataset, while in RL, at least in simulation, this is not the case, although with things like experience replay, one may think of it as collecting a dataset, but this dataset is not fixed and can actually change anyway." I will not post this as a formal answer until you clarify the type of answer you're looking for.  
  > **username_0**: @username_1 Although they are same, I am guessing that there may be some differences if we go too deep in to formal aspects. So. to keep them intact at all levels. I used two different names.  
  > **username_0**: If I always start interpreting experience in RL as the data in statistical AI, then will I get contradiction somewhere or not is the question.  
  > **username_2**: I do not understand this question, I think because you do not clarify anywahere what you mean by "treat like" from the title. Or "interpret" in the question body. Assuming there are some useful parallels between the two terms (and I think there are), what do you intend to **do** with that information? Are you looking for ways to transform between supervised learning and RL, or trying to understand the differences between RL experience and a supervised learning dataset?  
  > **username_0**: @username_2 trying to understand difference between RL experience and SL data.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-09-06. Score: 2
  
  The main similarity between reinforcement learning experience and supervised learning datasets, is that both consist of a set of records. These records are commonly expressed as vectors of numbers for use in the algorithms. In addition, reinforcement learning that uses neural networks (or other function approximation) will typically implement some variant of supervised learning internally.
  
  There are a few key differences between a prepared dataset for supervised learning and the experience in reinforcement learning. There are exceptions to these, but these are the usual case:
  
  * A supervised learning dataset has a fixed target value to learn by association, for each entry, e.g. a class or regression value. An individual reinforcement learning experience does not - the raw tuple of state, action, reward, next state $(s,a,r,s')$ must be processed in some way to obtain a useful training target value, and this processing is *not static*. When training in reinforcement learning for optimal control, even with historical experience, the target values must constantly be re-assessed.
  * Reinforcement learning is in part a design for actively collecting experience. There is no equivalent in supervised learning where the dataset is a given.
  * Reinforcement learning experience arrives in groups of $(s, a, r, s')$ - state, action, reward, next state - such groups of related data within a record are called *tuples*. The RL records are often correlated with each other, at least initially because each time step changes things only slightly, and that can be bad when combined with supervised learning which usually assumes uncorrelated data. In supervised learning you will often have a deliberate shuffling or randomisation of dataset order to protect against this. Experience replay in deep RL is a related idea to protect internal neural networks from being exposed to training samples in sequences with correlated values.
  
  It is possible to apply reinforcement learning to supervised learning problems in theory. You can do this by making the agent guess each labelled value as an action, and reward it with negative the loss from the supervised learning. This is generally a bad idea because it is very inefficient, and there is no matching concept of state transitions in the supervised learning problem (the agent cannot impact what state is next due to its guess). However, the fact that this is possible with very little modification to the reinforcement learning agent shows how general reinforcement learning is as a learning approach.
  
  The inverse is not really true, you cannot normally adjust a supervised learning algorithm so that it can solve a reinforcement learning problem from the given experience. There are some edge cases, such as when learning only from previous experience to assess an existing policy, or to learn a control algorithm which only cares about immediate reward. In which case you could use reinforcement learning theory to help construct a fixed dataset and give that to a supervised learning algorithm. However, by far the most common approach is to use supervised learning approaches (e.g. a neural network) as components of the agent, and rely on reinforcement learning to generate data for them on the fly.
  
  
  ---
  
  