Answers.Body: <blockquote>
  <p>However, if we can model the environment, why should we want to employ a model-free algorithm?</p>
  </blockquote>
  <p>Depends what you mean by &quot;model the environment&quot;. There are two kinds of model:</p>
  <ul>
  <li><p>Distribution model, which provides full access to a function like <span class="math-container">$p(r,s'|s,a)$</span>, the probability of observing reward <span class="math-container">$r$</span> and next state <span class="math-container">$s'$</span> given starting state <span class="math-container">$s$</span> and taking action <span class="math-container">$a$</span>.</p>
  </li>
  <li><p>Sample model, which can process a single step within the environment on demand. That should provide access to a function like <span class="math-container">$\text{step}(s,a)$</span> that returns a single <span class="math-container">$(r, s')$</span> pair based on running the environment forward from <span class="math-container">$(s,a)$</span>.</p>
  </li>
  </ul>
  <p>Which kind of model is available makes a difference to which kinds of model-based reinforcement-learning approaches will work. For instance a distribution model is required for dynamic programming, but MCTS may only need a sample model.</p>
  <p>A distribution model can be converted easily to a sample model, but the other way around is only possible approximately (by taking lots of samples).</p>
  <blockquote>
  <p>Isn't it better to have a model and expectation about the next reward and state?</p>
  </blockquote>
  <p>All things being equal, then yes having access to and using a model allows for better estimates, and opens up possibilities for various types of planning algorithm.</p>
  <p>However, even if you can build a model of an environment, there are costs:</p>
  <ul>
  <li><p>Writing a distribution model can be hard theoretically - it is often far easier to simulate an environment for a single step forward than calculate probability distributions for all outcomes of the same step.</p>
  </li>
  <li><p>The need for random access to arbitrary states can be expensive. A simulator that only needs to run forward from a single reset to a start state may run much more efficiently than one that needs to be set to multiple places in a tree of possibilities arbitrarily.</p>
  </li>
  </ul>
  <p>These costs are different for different environments. Setting a game like chess or go to arbitrary states is relatively cheap compared to a modern computer game or something that simulates a real-world physical system with lots of detail. Partially-observable states can be a major problem in both cases, since any random access model would need to account for all possible &quot;true&quot; states that lie behind whatever the agent is processing.</p>
  <p>So basically, at different levels of environment complexity, either the development cost of making a full model, or the evaluation cost of setting arbitrary states can make it more effective to use a model-free approach.</p>
  <p>Agents that learn to build their own approximate models of the environment are of great interest, as these potentially address the issue by getting the best of both world. Or possibly they could be tunable to use model-free vs model-based approaches in the most efficient manner depending on the relative costs and accuracy of each approach.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-04T17:30:47.770Z
Answers.Text: I'm not sure I understand why you differentiate between what you call a "distribution model" and a "sampling model". You always need something that returns you the next state $s'$ and reward $r$ for every state $s$ and action $a$ for any RL algorithm; so, if I understand correctly your terminology, you always need a "sampling model" anyway. It seems to me that a sampling model is just the distribution model, but without knowing the probabilities, i.e. just a way to sample from $p.$
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-04T17:36:49.337Z
Answers.Text: I wouldn't call it a sampling model, but _step/sampling function_ (as you do) to differentiate it from what we usually call "model" in RL (not sure if "sampling model" is used in the literature, I don't know). The step function may be stochastic, but we may not know exact how it computes. Or your "sampling model" would be kind of a simulation of the real environment?
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-04T17:49:40.120Z
Answers.Text: @nbro: Sample model and distribution model are terms used in e.g. Sutton & Barto. A distribution model can produce a sample model, but not the other way around (except via some statistical learning algorithm). Technically the book uses "sample model", not "sampl*ing* model", so I'll change my answer to match
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-04T17:55:21.023Z
Answers.Text: Ok. It's been a while since I read that book, and probably that term is only used in some chapters and maybe in the 2nd ed (I will check that).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-04T17:55:33.010Z
Answers.Text: @nbro: You will not need a sample model if you have access to the real environment. As some real environments are virtual, implemented in code, then sometimes that line is blurred. Anyway, with a real environment, the step function is not calling a model, it is calling the real instance of the thing.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-05T12:47:14.797Z
Answers.Text: I am kind of confused. The definition of the model necessitates transition and reward function. In this case, I am not quite sure what you mean by "real environment".
Answers.UserDisplayName: Ayska
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-05T12:54:39.833Z
Answers.Text: @NeilSlater Thanks for your answer. I see your point. Thus, if I have a physical dynamic system like the one that you generally can solve by linear programming or non-linear programming, I can actually create my model because I know the system dynamics and can write as an equation. Then, I need to use a policy or value iteration type solution only since I should employ dynamic programming methods.
Answers.UserDisplayName: Ayska
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-05T12:54:50.070Z
Answers.Text: A real environment is not a model. It is the real thing. This sometimes get lost conceptually because many environments are simulations, games etc that are inside the same computer as the agent. However, you would not write transition functions for a physical robot - the transition logic is what happens in the real world to the robot.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-05T12:58:28.957Z
Answers.Text: @NeilSlater So my transition function is getting after I created my solution and becoming with experiences?
Answers.UserDisplayName: Ayska
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-04T17:00:49.430Z
Answers.LastActivityDate: 2021-08-04T17:53:10.683Z
Answers.LastEditDate: 2021-08-04T17:53:10.683Z
Answers.UserDisplayName: Neil Slater
Body: <p>I am trying to understand the concept of model-free and model-based approaches. As far as I understand, having a model of the environment does not mean that an RL agent has to be model-based. It is about the policy. However, if we can model the environment, why should we want to employ a model-free algorithm? Isn't it better to have a model and expectation about the next reward and state? If you have a better understanding of all these, can you explain them to me as well?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-04T16:26:06.467Z
Comments.Text: I am not sure what you mean by "It is about the policy"? Could you please clarify?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-05T12:37:59.833Z
Comments.Text: I mean, even if we have a model, the main reason for having this model is to find the optimal policy. (If I am right!).
Comments.UserDisplayName: Ayska
Tags: <reinforcement-learning><ai-design><model-based-methods><model-free-methods>
Title: If we can model the environment, wouldn't be meaningless to use a model-free algorithm?
UserDisplayName: Ayska
ThreadText: # If we can model the environment, wouldn't be meaningless to use a model-free algorithm?
  
  (Asked by: username_0 on 2021-08-04)
  
  I am trying to understand the concept of model-free and model-based approaches. As far as I understand, having a model of the environment does not mean that an RL agent has to be model-based. It is about the policy. However, if we can model the environment, why should we want to employ a model-free algorithm? Isn't it better to have a model and expectation about the next reward and state? If you have a better understanding of all these, can you explain them to me as well?
  
  **Tags:** reinforcement-learning, ai-design, model-based-methods, model-free-methods
  
  > **Comments:**
  > **username_1**: I am not sure what you mean by "It is about the policy"? Could you please clarify?  
  > **username_0**: I mean, even if we have a model, the main reason for having this model is to find the optimal policy. (If I am right!).  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-08-04. Score: 2 | Accepted answer
  
  > However, if we can model the environment, why should we want to employ a model-free algorithm?
  
  Depends what you mean by "model the environment". There are two kinds of model:
  
  * Distribution model, which provides full access to a function like $p(r,s'|s,a)$, the probability of observing reward $r$ and next state $s'$ given starting state $s$ and taking action $a$.
  * Sample model, which can process a single step within the environment on demand. That should provide access to a function like $\text{step}(s,a)$ that returns a single $(r, s')$ pair based on running the environment forward from $(s,a)$.
  
  Which kind of model is available makes a difference to which kinds of model-based reinforcement-learning approaches will work. For instance a distribution model is required for dynamic programming, but MCTS may only need a sample model.
  
  A distribution model can be converted easily to a sample model, but the other way around is only possible approximately (by taking lots of samples).
  
  > Isn't it better to have a model and expectation about the next reward and state?
  
  All things being equal, then yes having access to and using a model allows for better estimates, and opens up possibilities for various types of planning algorithm.
  
  However, even if you can build a model of an environment, there are costs:
  
  * Writing a distribution model can be hard theoretically - it is often far easier to simulate an environment for a single step forward than calculate probability distributions for all outcomes of the same step.
  * The need for random access to arbitrary states can be expensive. A simulator that only needs to run forward from a single reset to a start state may run much more efficiently than one that needs to be set to multiple places in a tree of possibilities arbitrarily.
  
  These costs are different for different environments. Setting a game like chess or go to arbitrary states is relatively cheap compared to a modern computer game or something that simulates a real-world physical system with lots of detail. Partially-observable states can be a major problem in both cases, since any random access model would need to account for all possible "true" states that lie behind whatever the agent is processing.
  
  So basically, at different levels of environment complexity, either the development cost of making a full model, or the evaluation cost of setting arbitrary states can make it more effective to use a model-free approach.
  
  Agents that learn to build their own approximate models of the environment are of great interest, as these potentially address the issue by getting the best of both world. Or possibly they could be tunable to use model-free vs model-based approaches in the most efficient manner depending on the relative costs and accuracy of each approach.
  
  > **Comments:**
  > **username_2**: I'm not sure I understand why you differentiate between what you call a "distribution model" and a "sampling model". You always need something that returns you the next state $s'$ and reward $r$ for every state $s$ and action $a$ for any RL algorithm; so, if I understand correctly your terminology, you always need a "sampling model" anyway. It seems to me that a sampling model is just the distribution model, but without knowing the probabilities, i.e. just a way to sample from $p.$  
  > **username_2**: I wouldn't call it a sampling model, but _step/sampling function_ (as you do) to differentiate it from what we usually call "model" in RL (not sure if "sampling model" is used in the literature, I don't know). The step function may be stochastic, but we may not know exact how it computes. Or your "sampling model" would be kind of a simulation of the real environment?  
  > **username_1**: @username_2: Sample model and distribution model are terms used in e.g. Sutton & Barto. A distribution model can produce a sample model, but not the other way around (except via some statistical learning algorithm). Technically the book uses "sample model", not "sampl*ing* model", so I'll change my answer to match  
  > **username_2**: Ok. It's been a while since I read that book, and probably that term is only used in some chapters and maybe in the 2nd ed (I will check that).  
  > **username_1**: @username_2: You will not need a sample model if you have access to the real environment. As some real environments are virtual, implemented in code, then sometimes that line is blurred. Anyway, with a real environment, the step function is not calling a model, it is calling the real instance of the thing.  
  > **username_0**: I am kind of confused. The definition of the model necessitates transition and reward function. In this case, I am not quite sure what you mean by "real environment".  
  > **username_0**: @username_1 Thanks for your answer. I see your point. Thus, if I have a physical dynamic system like the one that you generally can solve by linear programming or non-linear programming, I can actually create my model because I know the system dynamics and can write as an equation. Then, I need to use a policy or value iteration type solution only since I should employ dynamic programming methods.  
  > **username_1**: A real environment is not a model. It is the real thing. This sometimes get lost conceptually because many environments are simulations, games etc that are inside the same computer as the agent. However, you would not write transition functions for a physical robot - the transition logic is what happens in the real world to the robot.  
  > **username_0**: @username_1 So my transition function is getting after I created my solution and becoming with experiences?  
  
  ---
  
  