Answers.Body: <blockquote>
    <p>I am aware that as the experience set grows the Central Limit Theorem
    will come into play and the distribution of experience will more
    accurately represent the true environment's state-actions-rewards
    distribution</p>
  </blockquote>
  
  <p>I believe here you mean the <a href="https://en.wikipedia.org/wiki/Law_of_large_numbers" rel="nofollow noreferrer">Law of Large Numbers</a> which states that for a large enough sample (<span class="math-container">$n \rightarrow \infty$</span>) the sample mean will converge to the true mean. The <a href="https://en.wikipedia.org/wiki/Central_limit_theorem" rel="nofollow noreferrer">central limit theorem (CLT)</a> states that if you take the sum/mean of a set of <em>independent</em> random variables then the distribution of this new RV will be approximately normal as <span class="math-container">$n \rightarrow \infty$</span>. </p>
  
  <blockquote>
    <p>before this happens is sampled based planning still useful</p>
  </blockquote>
  
  <p>As you mentioned, if you had only 5 full episodes of experience to choose from, then this likely would not represent true underlying distributions and so the approximations would not be good -- of course, this will depend on how complex your MDP is, if you had a trivially simple one then it may be enough to represent it well. As David Silver says in his lecture, then one of the disadvantages of planning with a model is that you introduce another set of uncertainty, mainly from when you approximate the properties of the model. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-31T13:29:59.043Z
Answers.Text: Much appreciated. I did mean the Law of Large Numbers, thanks for the correction.
  
  So it all seems a trade-off between MDP complexity and the size of experience set.
Answers.UserDisplayName: BlueTurtle
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-31T16:55:59.367Z
Answers.Text: yes, as you are just learning the model through supervised learning methods, the same rules apply i.e. the complexity of your problem will determine how much data you need to balance the bias/variance trade off
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-31T11:23:22.013Z
Answers.LastActivityDate: 2020-06-01T14:38:38.287Z
Answers.LastEditDate: 2020-06-01T14:38:38.287Z
Answers.UserDisplayName: David
Body: <p>From the <a href="https://www.youtube.com/watch?v=ItMutbeOHtc" rel="nofollow noreferrer">David Silver's lecture 8: Integrating Learning and Planning</a> - based on Sutton and Barto - he talks about using sample-based planning to use our model to take a sample of a state and then use model-free planning, such as Monte Carlo, etc, to run the trajectory and observe the reward. He goes on to say that this effectively gives us infinite data from only a few actual experiences.</p>
  
  <p>However, if we only experience a handful of true state-action-rewards and then start sampling to learn more then we will surely end up with a skewed result, e.g., If I have 5 experiences but then create 10000 samples (as he says, infinite data). I am aware that as the experience set grows the Central Limit Theorem will come into play and the distribution of experience will more accurately represent the true environment's state-actions-rewards distribution but before this happens is sampled based planning still useful?</p>
  
Tags: <reinforcement-learning><planning>
Title: Is the distribution of state-action pairs from sample based planning accurate for small experience sets?
UserDisplayName: BlueTurtle
ThreadText: # Is the distribution of state-action pairs from sample based planning accurate for small experience sets?
  
  (Asked by: username_0 on 2020-05-30)
  
  From the [David Silver's lecture 8: Integrating Learning and Planning](https://www.youtube.com/watch?v=ItMutbeOHtc) - based on Sutton and Barto - he talks about using sample-based planning to use our model to take a sample of a state and then use model-free planning, such as Monte Carlo, etc, to run the trajectory and observe the reward. He goes on to say that this effectively gives us infinite data from only a few actual experiences.
  
  However, if we only experience a handful of true state-action-rewards and then start sampling to learn more then we will surely end up with a skewed result, e.g., If I have 5 experiences but then create 10000 samples (as he says, infinite data). I am aware that as the experience set grows the Central Limit Theorem will come into play and the distribution of experience will more accurately represent the true environment's state-actions-rewards distribution but before this happens is sampled based planning still useful?
  
  **Tags:** reinforcement-learning, planning
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-05-31. Score: 2 | Accepted answer
  
  > I am aware that as the experience set grows the Central Limit Theorem
  > will come into play and the distribution of experience will more
  > accurately represent the true environment's state-actions-rewards
  > distribution
  
  I believe here you mean the [Law of Large Numbers](https://en.wikipedia.org/wiki/Law_of_large_numbers) which states that for a large enough sample ($n \rightarrow \infty$) the sample mean will converge to the true mean. The [central limit theorem (CLT)](https://en.wikipedia.org/wiki/Central_limit_theorem) states that if you take the sum/mean of a set of *independent* random variables then the distribution of this new RV will be approximately normal as $n \rightarrow \infty$.
  
  > before this happens is sampled based planning still useful
  
  As you mentioned, if you had only 5 full episodes of experience to choose from, then this likely would not represent true underlying distributions and so the approximations would not be good -- of course, this will depend on how complex your MDP is, if you had a trivially simple one then it may be enough to represent it well. As David Silver says in his lecture, then one of the disadvantages of planning with a model is that you introduce another set of uncertainty, mainly from when you approximate the properties of the model.
  
  > **Comments:**
  > **username_0**: Much appreciated. I did mean the Law of Large Numbers, thanks for the correction.  So it all seems a trade-off between MDP complexity and the size of experience set.  
  > **username_1**: yes, as you are just learning the model through supervised learning methods, the same rules apply i.e. the complexity of your problem will determine how much data you need to balance the bias/variance trade off  
  
  ---
  
  