Answers.Body: <p>If you already have some transition tuples then you can train a model to predict environment dynamics using these. However, you should be careful that your pre-gathered data is diverse enough to 'cover' enough of the state/action space so that your model remains accurate. For instance, when you start training your agent it will likely start to see more of the state space than it did at the start of training (imagine playing Atari, initially your agent will die quickly but as it gets better episodes will get longer) so you would need to make sure you have data for these states that appear late in episodes, otherwise your model will just be overfitting to the start of the episode and will give a poor performance on these other states, thus slowing down or even prohibiting learning of an optimal policy.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-07T09:50:42.847Z
Answers.Text: Thank you, your issue is clear for me. So can I use this historical data as an input in model-base rl or I need some additional modeling techniques to estimate stansition?
Answers.UserDisplayName: user46045
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-07T09:53:23.700Z
Answers.Text: you could try your model with your historical data, at the least it would give it a good 'initialisation'. you could then update your model once you have some new data that the agent collects.
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-07T09:57:45.887Z
Answers.Text: Could you advise me some good library or tutorial for coding purpose in model-free Rl?
Answers.UserDisplayName: user46045
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-07T09:59:08.360Z
Answers.Text: if you want to learn model-free RL then I would look for some medium articles on DQN, they are usually a good resource with well explained code.
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-07T10:00:25.987Z
Answers.Text: Sorry, I mean model-based. It is my mistake
Answers.UserDisplayName: user46045
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-07T10:03:17.103Z
Answers.Text: I'm afraid not, I have never worked with model-based methods before. You could look at some papers and see if the authors have a public GitHub repo.
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-21T04:55:53.060Z
Answers.Text: cs285 has a comprehensive lecture on model-based RL @user46045. [lecture video](https://youtu.be/6JDfrPRhexQ?list=PLkFD6_40KJIwhWJpGazJ9VSj9CFMkb79A). [MBRL notes](http://rail.eecs.berkeley.edu/deeprlcourse/static/slides/lec-11.pdf), [Practice code](http://rail.eecs.berkeley.edu/deeprlcourse/static/homeworks/hw4.pdf). I've added these a bit late, but they might help someone else later.
Answers.UserDisplayName: mugoh
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-07T09:37:02.970Z
Answers.LastActivityDate: 2021-04-07T09:37:02.970Z
Answers.UserDisplayName: David
Body: <p>I want to build model-based RL. I am wondering about the process of building the model.</p>
  <p>If I already have data, from real experience:</p>
  <ul>
  <li><span class="math-container">$S_1, a \rightarrow R,S_2$</span></li>
  <li><span class="math-container">$S_2, a \rightarrow R,S_3$</span></li>
  </ul>
  <p>Can I use this information, to build model-based RL? Or it is necessary that the agent directly interact with the environment (I mean the same above-mentioned data should be provided by the agent)?</p>
  
Tags: <reinforcement-learning><model-based-methods>
Title: How does a model based agent learn the model?
UserDisplayName: user46045
ThreadText: # How does a model based agent learn the model?
  
  (Asked by: username_0 on 2021-04-07)
  
  I want to build model-based RL. I am wondering about the process of building the model.
  
  If I already have data, from real experience:
  
  * $S\_1, a \rightarrow R,S\_2$
  * $S\_2, a \rightarrow R,S\_3$
  
  Can I use this information, to build model-based RL? Or it is necessary that the agent directly interact with the environment (I mean the same above-mentioned data should be provided by the agent)?
  
  **Tags:** reinforcement-learning, model-based-methods
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-04-07. Score: 1 | Accepted answer
  
  If you already have some transition tuples then you can train a model to predict environment dynamics using these. However, you should be careful that your pre-gathered data is diverse enough to 'cover' enough of the state/action space so that your model remains accurate. For instance, when you start training your agent it will likely start to see more of the state space than it did at the start of training (imagine playing Atari, initially your agent will die quickly but as it gets better episodes will get longer) so you would need to make sure you have data for these states that appear late in episodes, otherwise your model will just be overfitting to the start of the episode and will give a poor performance on these other states, thus slowing down or even prohibiting learning of an optimal policy.
  
  > **Comments:**
  > **username_0**: Thank you, your issue is clear for me. So can I use this historical data as an input in model-base rl or I need some additional modeling techniques to estimate stansition?  
  > **username_1**: you could try your model with your historical data, at the least it would give it a good 'initialisation'. you could then update your model once you have some new data that the agent collects.  
  > **username_0**: Could you advise me some good library or tutorial for coding purpose in model-free Rl?  
  > **username_1**: if you want to learn model-free RL then I would look for some medium articles on DQN, they are usually a good resource with well explained code.  
  > **username_0**: Sorry, I mean model-based. It is my mistake  
  > **username_1**: I'm afraid not, I have never worked with model-based methods before. You could look at some papers and see if the authors have a public GitHub repo.  
  > **username_2**: cs285 has a comprehensive lecture on model-based RL @username_0. [lecture video](https://youtu.be/6JDfrPRhexQ?list=PLkFD6_40KJIwhWJpGazJ9VSj9CFMkb79A). [MBRL notes](http://rail.eecs.berkeley.edu/deeprlcourse/static/slides/lec-11.pdf), [Practice code](http://rail.eecs.berkeley.edu/deeprlcourse/static/homeworks/hw4.pdf). I've added these a bit late, but they might help someone else later.  
  
  ---
  
  