Answers.Body: <p>Model-based methods (such as value or <a href="http://www.incompleteideas.net/book/first/ebook/node43.html" rel="nofollow noreferrer">policy iteration</a>) use a <em>model</em> of the environment, which is usually represented as a <a href="https://en.wikipedia.org/wiki/Markov_decision_process" rel="nofollow noreferrer">Markov decision process</a>. More specifically, the model consists of the transition and reward functions of the Markov decision process, which should represent the <em>dynamics</em> of the environment. For example, in <a href="http://www.incompleteideas.net/book/first/ebook/node43.html" rel="nofollow noreferrer">policy iteration</a>, the rewards (used to estimate the policy or value functions) are not the result of the interaction with the environment but given by the MDP (the model of the environment), so the decisions are made according to the reward function (and the transition function) of the MDP that represents the dynamics of the environment. Model-based methods are not (usually) dependent on past actions. For example, policy iteration converges to the optimal policy independently of the initial values of the states, the initial policy or the order of iteration through the states.</p>
  
  <p>Monte Carlo methods do not <strong>use</strong> such a model (the MDP), even though the assumption that the environment can be represented as an MDP is (often implicitly) made (and the MDP might actually be available). In the case of Monte Carlo methods, all estimates are solely based on the interaction with the environment. In general, Monte Carlo methods are based on sampling (or random) operations. In the case of reinforcement learning, they sample the environment. The samples are the rewards that are obtained when certain actions are taken from certain states.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-27T13:52:37.050Z
Answers.Text: Thanks for the explanation. About the Monte Carlo(MC) not being MDP, is this always true? Or MC could be MDP as well? I am not sure if this makes sense, so far I had this intuition that everything is MDP.
Answers.UserDisplayName: tausif
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-27T13:57:13.703Z
Answers.Text: @tausif: Everything in RL *is* based on MDP theory, including Monte Carlo control. The difference is that for model-based approaches you need the MDP or a decent proxy for it, that the agent can query directly (independently of having experience). There are broadly two types of model and two types of model-dependent RL - sampling models and distribution models - having the full MDP gives you a distribution model. I think the last paragraph here could make it clearer that there still is an assumed MDP for the Monte Carlo methods to work, just you do not need *access* to it
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-27T14:07:52.843Z
Answers.Text: @NeilSlater I would not say that everything in RL is based on MDP, unless by MDPs you mean all kinds of MDPs, including e.g. POMDPs.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-27T14:46:32.720Z
Answers.Text: @tausif As Neil stated, the common MC methods (often _implicitly_) assume that the environment can be represented as an MDP. However, this does not mean that they use or even know the MDP.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-07T10:03:52.817Z
Answers.Text: MCTS is model based when used to play chess and similar games. Montecarlo refers to probability and random usage
Answers.UserDisplayName: Raul Lapeira Herrero
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-27T13:01:27.880Z
Answers.LastActivityDate: 2019-07-28T22:21:24.200Z
Answers.LastEditDate: 2019-07-28T22:21:24.200Z
Answers.UserDisplayName: nbro
Body: <p>I was going through an article where it is mentioned:</p>
  
  <blockquote>
    <p>The Monte-Carlo methods require only knowledge base (history/past experiences)—sample sequences of (states, actions and rewards) from the interaction with the environment, and no actual model of the environment.</p>
  </blockquote>
  
  <p>Aren't the model-based method dependent on past sequences? How is Monte Carlo is different than?</p>
  
Tags: <reinforcement-learning><comparison><monte-carlo-methods><model-based-methods>
Title: How is Monte Carlo different from model-based methods?
UserDisplayName: tausif
ThreadText: # How is Monte Carlo different from model-based methods?
  
  (Asked by: username_0 on 2019-07-27)
  
  I was going through an article where it is mentioned:
  
  > The Monte-Carlo methods require only knowledge base (history/past experiences)—sample sequences of (states, actions and rewards) from the interaction with the environment, and no actual model of the environment.
  
  Aren't the model-based method dependent on past sequences? How is Monte Carlo is different than?
  
  **Tags:** reinforcement-learning, comparison, monte-carlo-methods, model-based-methods
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-07-27. Score: 0 | Accepted answer
  
  Model-based methods (such as value or [policy iteration](http://www.incompleteideas.net/book/first/ebook/node43.html)) use a *model* of the environment, which is usually represented as a [Markov decision process](https://en.wikipedia.org/wiki/Markov_decision_process). More specifically, the model consists of the transition and reward functions of the Markov decision process, which should represent the *dynamics* of the environment. For example, in [policy iteration](http://www.incompleteideas.net/book/first/ebook/node43.html), the rewards (used to estimate the policy or value functions) are not the result of the interaction with the environment but given by the MDP (the model of the environment), so the decisions are made according to the reward function (and the transition function) of the MDP that represents the dynamics of the environment. Model-based methods are not (usually) dependent on past actions. For example, policy iteration converges to the optimal policy independently of the initial values of the states, the initial policy or the order of iteration through the states.
  
  Monte Carlo methods do not **use** such a model (the MDP), even though the assumption that the environment can be represented as an MDP is (often implicitly) made (and the MDP might actually be available). In the case of Monte Carlo methods, all estimates are solely based on the interaction with the environment. In general, Monte Carlo methods are based on sampling (or random) operations. In the case of reinforcement learning, they sample the environment. The samples are the rewards that are obtained when certain actions are taken from certain states.
  
  > **Comments:**
  > **username_0**: Thanks for the explanation. About the Monte Carlo(MC) not being MDP, is this always true? Or MC could be MDP as well? I am not sure if this makes sense, so far I had this intuition that everything is MDP.  
  > **username_2**: @username_0: Everything in RL *is* based on MDP theory, including Monte Carlo control. The difference is that for model-based approaches you need the MDP or a decent proxy for it, that the agent can query directly (independently of having experience). There are broadly two types of model and two types of model-dependent RL - sampling models and distribution models - having the full MDP gives you a distribution model. I think the last paragraph here could make it clearer that there still is an assumed MDP for the Monte Carlo methods to work, just you do not need *access* to it  
  > **username_1**: @username_2 I would not say that everything in RL is based on MDP, unless by MDPs you mean all kinds of MDPs, including e.g. POMDPs.  
  > **username_1**: @username_0 As username_2 stated, the common MC methods (often _implicitly_) assume that the environment can be represented as an MDP. However, this does not mean that they use or even know the MDP.  
  > **username_3**: MCTS is model based when used to play chess and similar games. Montecarlo refers to probability and random usage  
  
  ---
  
  