Answers.Body: <blockquote>
  <p>What's the difference between model-free and model-based reinforcement learning?</p>
  </blockquote>
  <p>In Reinforcement Learning, the terms &quot;model-based&quot; and &quot;model-free&quot; do <em>not</em> refer to the use of a neural network or other statistical learning model to predict values, or even to predict next state (although the latter may be used as part of a model-based algorithm and be called a &quot;model&quot; regardless of whether the algorithm is model-based or model-free).</p>
  <p>Instead, the term refers strictly as to whether, whilst during learning or acting, the agent uses predictions of the environment response. The agent can use a single prediction from the model of next reward and next state (a sample), or it can ask the model for the <em>expected</em> next reward, or the full <em>distribution</em> of next states and next rewards. These predictions can be provided entirely outside of the learning agent - e.g. by computer code that understands the rules of a dice or board game. Or they can be learned by the agent, in which case they will be approximate.</p>
  <p>Just because there is a model of the environment implemented, does not mean that a RL agent is &quot;model-based&quot;. To qualify as &quot;model-based&quot;, the learning algorithms have to explicitly reference the model:</p>
  <ul>
  <li><p>Algorithms that purely sample from experience such as Monte Carlo Control, SARSA, Q-learning, Actor-Critic are &quot;model free&quot; RL algorithms. They rely on real samples from the environment and never use generated predictions of next state and next reward to alter behaviour (although they might sample from experience memory, which is close to being a model).</p>
  </li>
  <li><p>The archetypical model-based algorithms are Dynamic Programming (Policy Iteration and Value Iteration) - these all use the model's predictions or distributions of next state and reward in order to calculate optimal actions. Specifically in Dynamic Programming, the model must provide state transition probabilities, and expected reward from any state, action pair. Note this is rarely a learned model.</p>
  </li>
  <li><p>Basic TD learning, using state values only, must also be model-based in order to work as a control system and pick actions. In order to pick the best action, it needs to query a model that predicts what will happen on each action, and implement a policy like <span class="math-container">$\pi(s) = \text{argmax}_a \sum_{s',r} p(s',r|s,a)(r + v(s'))$</span> where <span class="math-container">$p(s',r|s,a)$</span> is the probability of receiving reward <span class="math-container">$r$</span> and next state <span class="math-container">$s'$</span> when taking action <span class="math-container">$a$</span> in state <span class="math-container">$s$</span>. That function <span class="math-container">$p(s',r|s,a)$</span> is essentially the model.</p>
  </li>
  </ul>
  <p>The RL literature differentiates between &quot;model&quot; as a model of the environment for &quot;model-based&quot; and &quot;model-free&quot; learning, and use of statistical learners, such as neural networks.</p>
  <p>In RL, neural networks are often employed to learn and generalise value functions, such as the Q value which predicts total return (sum of discounted rewards) given a state and action pair. Such a trained neural network is often called a &quot;model&quot; in e.g. supervised learning. However, in RL literature, you will see the term &quot;function approximator&quot; used for such a network to avoid ambiguity.</p>
  <blockquote>
  <p>It seems to me that any model-free learner, learning through trial and error, could be reframed as model-based.</p>
  </blockquote>
  <p>I think here you are using the general understanding of the word &quot;model&quot; to include any structure that makes useful predictions. That would apply to e.g. table of Q values in SARSA.</p>
  <p>However, as explained above, that's not how the term is used in RL. So although your understanding that RL builds useful internal representations is correct, you are not technically correct that this can be used to re-frame between &quot;model-free&quot; as &quot;model-based&quot;, because those terms have a very specific meaning in RL.</p>
  <blockquote>
  <p>In that case, when would model-free learners be appropriate?</p>
  </blockquote>
  <p>Generally with current state of art in RL, if you don't have an accurate model provided as part of the problem definition, then model-free approaches are often superior.</p>
  <p>There is lots of interest in agents that build predictive models of the environment, and doing so as a &quot;side effect&quot; (whilst still being a model-free algorithm) can still be useful - it may regularise a neural network or help discover key predictive features that can also be used in policy or value networks. However, model-based agents that learn their own models for planning have a problem that inaccuracy in these models can cause instability (the inaccuracies multiply the further into the future the agent looks). Some promising inroads are being made using <a href="https://www.deepmind.com/blog/agents-that-imagine-and-plan" rel="noreferrer">imagination-based agents</a> and/or mechanisms for deciding when and how much to trust the learned model during planning.</p>
  <p>Right now (in 2018), if you have a real-world problem in an environment without an explicit known model at the start, then the safest bet is to use a model-free approach such as DQN or A3C. That may change as the field is moving fast and new more complex architectures could well be the norm in a few years.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-01T20:44:17.677Z
Answers.Text: A small correction, normally the terms "model based" or "model free" are not used for planning algorithms such as MCTS. It is only used to classify learning algorithms.
Answers.UserDisplayName: Miguel Saraiva
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-01T20:50:56.327Z
Answers.Text: @MiguelSaraiva: I'm not 100% certain about that, but have removed the reference to MCTS. Out of interest, where would you place DynaQ regarding this limitation of the usage of the terms? I think it becomes tricky, when the algorithms all share such a common view of the MDP model and improving policies, to tell where the boundaris are between planning and learning.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-01T20:58:26.603Z
Answers.Text: I could be wrong, I'm a novice in the area. I just remember a teacher from the field making that comment after I had done the same remark.
Answers.UserDisplayName: Miguel Saraiva
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-31T00:37:53.543Z
Answers.Text: @NeilSlater, what would you recommend for an environment which is fully observable, but also gives back a model of it -- the exact reward and transition functions are known. I think there could be a benefit in using something beyond  DQN or A3C to take advantage of that knowledge. I cannot use dynamic programming since state-space is too large. thanks
Answers.UserDisplayName: d56
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-31T08:40:32.680Z
Answers.Text: @d56: I think that could be a new question on the site. In brief, you can still use most RL methods, and add *planning* to refine estimates of return values.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-29T09:07:39.247Z
Answers.Text: Thank you for informative answer! I want to clarify one thing, for model-free algorithms you've said they "never use generated predictions of next state and next reward to alter behaviour" however in Q-learning agent may take greedy action(isn't this generated?) or random action even though it updates Q-value assuming agent took greedy action, isn't greedy action generated since it is not actually taken by the agent?
Answers.UserDisplayName: haneulkim
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-29T15:03:11.327Z
Answers.Text: @haneulkim It is the prediction of next reward and state that are part of a model of the environment. The action is treated separately, using a policy of some kind, and yes all learning agents must have a method to do that somehow. The difference is important: An agent can control its action choice, but cannot control any other aspect of the environment rules. When looking at the difference between action taken (exploring behaviour policy) and the value update (greedy target policy), it doesn't matter whether you consider this "generated", it has nothing to do with the model of the environment
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-31T11:26:45.027Z
Answers.Text: @NeilSlater You say that "_if you have a real-world problem in an environment without an explicit known model at the start, then the safest bet is to use a model-free approach such as_ ". You mean that you should use real world simulations?  usually, when using a simulation even if the "learning agent" does not use the model, the model is necessary -and present-in order to excute the script and study the algorithm performance. So, apparently it is correct to say that in real world problem not only the "learning algorithm" is the problem.
Answers.UserDisplayName: Hermes Morales
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-31T11:26:57.320Z
Answers.Text: Continuation: You also need to construct a correct "real world simulation", except if you can experiment in the real world.  In this sense, RL is not model free. Am I wright?
Answers.UserDisplayName: Hermes Morales
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-31T13:11:27.670Z
Answers.Text: @HermesMorales There is a complex relationship between models, simulation and planning, in terms of when you might consider that you are using one or the other. One important detail is whether you have a sampling model or a distribution model. Model-free methods are often paired with simulations which are effectively sampling models. If the end goal is to then use the trained agent in a real environment, you might consider the training session as planning
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-02T03:39:21.500Z
Answers.Text: Other sources suggest TD-learning is model-free though.
Answers.UserDisplayName: Sam
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-02T09:37:00.227Z
Answers.Text: @Sam: Yes TD learning is itself model-free. It doesn't *require* any model. However, it is often used in situations where there is a model in the combined system. E.g. DynaQ.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-02T09:53:14.987Z
Answers.Text: @NeilSlater so is the statement 'Basic TD learning, using state values only, must also be model-based in order to work as a control system and pick actions.' still correct (I was confused by it)?
Answers.UserDisplayName: Sam
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-02T11:43:45.337Z
Answers.Text: @Sam - the learning system in that case must be model-based, yes. Without a model, TD learning using state values cannot make decisions. You cannot run value-based TD learning in a control scenario otehrwise, which is why you would typically use SARSA or Q learning (which are TD learning on action values) if you want a model-free TD learner. TD on state values still works model-free in predicion scenarios though.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-06-13T12:47:50.293Z
Answers.LastActivityDate: 2022-06-11T11:26:36.480Z
Answers.LastEditDate: 2022-06-11T11:26:36.480Z
Answers.UserDisplayName: Neil Slater
Answers.Body: <p>Model-based reinforcement learning has an agent try to understand the world and create a model to represent it. Here the model is trying to capture 2 functions, the transition function from states <span class="math-container">$T$</span> and the reward function <span class="math-container">$R$</span>. From this model, the agent has a reference and can plan accordingly. </p>
  
  <p>However, it is not necessary to learn a model, and the agent can instead learn a policy directly using algorithms like Q-learning or policy gradient. </p>
  
  <p>A simple check to see if an RL algorithm is model-based or model-free is: </p>
  
  <p><em>If, after learning, the agent can make predictions about what the next state and reward will be before it takes each action, it's a model-based RL algorithm.</em></p>
  
  <p>If it can't, then it’s a model-free algorithm.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-07T07:10:47.120Z
Answers.Text: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/105284/discussion-on-answer-by-jaden-travnik-whats-the-difference-between-model-free-a).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2017-11-08T14:07:08.093Z
Answers.LastActivityDate: 2019-02-16T19:08:43.947Z
Answers.LastEditDate: 2019-02-16T19:08:43.947Z
Answers.UserDisplayName: Jaden Travnik
Answers.Body: <p>In reinforcement learning (RL), there is an <em>agent</em> which interacts with an <em>environment</em> (in time steps). At each time step, the agent decides and executes an <em>action</em>, <span class="math-container">$a$</span>, on an environment, and the environment responds to the agent by moving from the current <em>state</em> (of the environment), <span class="math-container">$s$</span>, to the next state (of the environment), <span class="math-container">$s'$</span>, and by emitting a scalar signal, called the <em>reward</em>, <span class="math-container">$r$</span>.  In principle, this interaction can continue forever or until e.g. the agent dies.</p>
  
  <p>The main goal of the agent is to collect the largest amount of reward "in the long run". To do that, the agent needs to find an optimal policy (roughly, the optimal strategy to behave in the environment). In general, a policy is a function which, given a current state of the environment, outputs an action (or a probability distribution over actions, if the policy is <em>stochastic</em>) to execute in the environment. A policy can thus be thought of as the "strategy" used by the agent to behave in this environment. An optimal policy (for a given environment) is a policy which, if followed, will make the agent collect the largest amount of reward in the long run (which is the goal of the agent). In RL, we are thus interested in finding optimal policies.</p>
  
  <p>The environment can be deterministic (that is, roughly, the same action in the same state leads to the same next state, for all time steps) or stochastic (or non-deterministic), that is, if the agent takes an action in a certain state, the resulting next state of the environment might not necessarily always be the same: there is a probability that it will be a certain state or another. Of course, these uncertainties will make the task of finding the optimal policy harder.</p>
  
  <p>In RL, the problem is often mathematically formulated as a <a href="https://en.wikipedia.org/wiki/Markov_decision_process" rel="noreferrer">Markov decision process</a> (MDP). A MDP is a way of representing the "dynamics" of the environment, that is, the way the environment will react to the possible actions the agent might take, at a given state. More precisely, an MDP is equipped with a <strong>transition function</strong> (or "transition model"), which is a function that, given the current state of the environment and an action (that the agent might take), outputs a probability of moving to any of the next states. A <strong>reward function</strong> is also associated with an MDP. Intuitively, the reward function outputs a reward, given the current state of the environment (and, possibly, an action taken by the agent and the next state of the environment). Collectively, the transition and reward functions are often called the <strong>model</strong> of environment. To conclude, the MDP is the problem and the solution to the problem is a policy. Furthermore, the "dynamics" of the environment are governed by the transition and reward functions (that is, the "model"). </p>
  
  <p>However, we often do not have the MDP, that is, we do not have the transition and reward functions (of the MDP associated the environment). Hence, we cannot estimate a policy from the MDP, because it is unknown. Note that, in general, if we had the transition and reward functions of the MDP associated with the environment, we could exploit them and retrieve an optimal policy (using dynamic programming algorithms).</p>
  
  <p>In the absence of these functions (that is, when the MDP is unknown), to estimate the optimal policy, the agent needs to interact with environment and observe the responses of the environment. This is often referred to as the "reinforcement learning problem", because the agent will need to estimate a policy by <em>reinforcing</em> its beliefs about the dynamics of the environment. Over time, the agent starts to understand how the environment responds to its actions, and it can thus start to estimate the optimal policy. Thus, in the RL problem, the agent estimates the optimal policy to behave in an unknown (or partially known) environment by interacting with it (using a "trial-and-error" approach).</p>
  
  <p>In this context, a <strong>model-based</strong> algorithm is an algorithm that uses the transition function (and the reward function) in order to estimate the optimal policy. The agent might have access only to an approximation of the transition function and reward functions, which can be learned by the agent while it interacts with the environment or it can be given to the agent (e.g. by another agent). In general, in a model-based algorithm, the agent can potentially predict the dynamics of the environment (during or after the learning phase), because it has an estimate of the transition function (and reward function). However, note that the transition and reward functions that the agent uses in order to improve its estimate of the optimal policy might just be approximations of the "true" functions. Hence, the optimal policy might never be found (because of these approximations).</p>
  
  <p>A <strong>model-free</strong> algorithm is an algorithm that estimates the optimal policy without using or estimating the dynamics (transition and reward functions) of the environment. In practice, a model-free algorithm either estimates a "value function" or the "policy" directly from experience (that is, the interaction between the agent and environment), without using neither the transition function nor the reward function. A value function can be thought of as a function which evaluates a state (or an action taken in a state), for all states. From this value function, a policy can then be derived.</p>
  
  <p>In practice, one way to distinguish between model-based or model-free algorithms is to look at the algorithms and see if they use the transition or reward function.</p>
  
  <p>For instance, let's look at the main update rule in the <em>Q-learning algorithm</em>:</p>
  
  <p><span class="math-container">$$Q(S_t, A_t) \leftarrow Q(S_t, A_t) + \alpha (R_{t+1} + \gamma \max_{a}Q(S_{t+1}, a) - Q(S_t, A_t))$$</span></p>
  
  <p>As we can see, this update rule does not use any probabilities defined by the MDP. Note: <span class="math-container">$R_{t+1}$</span> is just the reward that is obtained at the next time step (after taking the action), but it is not necessarily known beforehand. So, Q-learning is a model-free algorithm.</p>
  
  <p>Now, let's look at the main update rule of the <em>policy improvement</em> algorithm:</p>
  
  <p><span class="math-container">$$Q(s,a) \leftarrow  \sum_{s' \in \mathcal{S}, r\in\mathcal{R}}p(s',r|s,a)(r+\gamma V(s'))$$</span></p>
  
  <p>We can immediately observe it uses <span class="math-container">$p(s',r|s,a)$</span>, a probability defined by the MDP model. So, <em>policy iteration</em> (a dynamic programming algorithm), which uses the policy improvement algorithm, is a model-based algorithm.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-14T07:28:16.577Z
Answers.Text: Already upvoted. Very well explained sir. Just a confusion, that why can't we just learn a reward and transition function associated with environment and then find optimal policy by dynamic learning. And if we are given this environment model by another agent then why can't we just apply dynamic learning to find optimal policy?
Answers.UserDisplayName: user_3pij
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-10T02:31:57.507Z
Answers.Text: @user_3pij In principle, one can do what you are suggesting. However, as I say in my answer, if you first need to estimate the transition and reward functions, then you necessarily introduce more approximations in your calculations, which may not be desirable.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-15T18:02:48.127Z
Answers.Text: When you are comparing the equations of update rules, I don't understand why the model-free still uses $R_{t+1}$. You explained that this is not necessarily known beforehand, but then how is it eventually known? What are the starting conditions of this reward if we don't have a model?
Answers.UserDisplayName: mesllo
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-17T22:04:06.533Z
Answers.Text: @mesllo What I mean by "not known beforehand" is that $R_{t+1}$ is only known after you have taken the action $A_t$. $R_{t+1}$ is determined by the reward function, if $p(s', r \mid s, a)$ is not known. Note that the environment still behaves according to $p$ and the reward function, but $p$ may not be known.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-11-07T11:35:11.767Z
Answers.LastActivityDate: 2019-02-27T21:52:40.953Z
Answers.LastEditDate: 2019-02-27T21:52:40.953Z
Answers.UserDisplayName: nbro
Answers.Body: <p>Although there are several good answers, I want to add this paragraph from <a href="http://incompleteideas.net/book/bookdraft2017nov5.pdf" rel="noreferrer">Reinforcement Learning: An Introduction</a>, page 303, for a more psychological view on the difference.</p>
  
  <blockquote>
    <p>The distinction between model-free and model-based reinforcement learning algorithms
    corresponds to the distinction psychologists make between habitual and goal-directed
    control of learned behavioral patterns. Habits are behavior patterns triggered by appropriate
    stimuli and then performed more-or-less automatically. Goal-directed behavior,
    according to how psychologists use the phrase, is purposeful in the sense that it is controlled
    by knowledge of the value of goals and the relationship between actions and their
    consequences. Habits are sometimes said to be controlled by antecedent stimuli, whereas
    goal-directed behavior is said to be controlled by its consequences (Dickinson, 1980,
    1985). Goal-directed control has the advantage that it can rapidly change an animal’s
    behavior when the environment changes its way of reacting to the animal’s actions. While
    habitual behavior responds quickly to input from an accustomed environment, it is unable
    to quickly adjust to changes in the environment.</p>
  </blockquote>
  
  <p>It keeps going from there, and has a nice example afterwards.</p>
  
  <p>I think the main point that was not always explained in the other answers, is that in a model-free approach you still need some kind of environment to tell you what is the reward associated with your action. The big difference is that you do NOT need to store any information about the model. You give the environment your chosen action, you update your estimated policy, and you forget about it. On the other hand, in model-based approaches, you either need to know the state transitions history as in Dynamic Programming, or you need to be able to calculate all possible next states and associated rewards, from the present state.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-01T21:24:59.840Z
Answers.LastActivityDate: 2019-12-01T21:33:49.050Z
Answers.LastEditDate: 2019-12-01T21:33:49.050Z
Answers.UserDisplayName: Miguel Saraiva
Answers.Body: <h2>Model-Free RL</h2>
  <p>In Model-Free RL, the agent does not have access to a model of the environment. By environment I mean a function which predicts state transition and rewards.</p>
  <p>As of the time of writing, model-free methods are more popular and have been researched extensively.</p>
  <h2>Model-Based RL</h2>
  <p>In Model-Based RL, the agent has access to a model of the environment.</p>
  <p>Main advantage is that this allows the agent to plan ahead by thinking ahead. Agents distill the results from planning ahead into a learned policy. A famous example of Model-Based RL is <a href="https://arxiv.org/abs/1712.01815" rel="nofollow noreferrer">AlphaZero</a>.</p>
  <p>The main downside is that many times a ground-truth representation of the environment is not usually available.</p>
  <hr />
  <p>Below is a non-exhaustive taxonomy of RL algorithms, which may help you to visualize better the RL landscape.</p>
  <p><a href="https://i.sstatic.net/oxKqm.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/oxKqm.png" alt="enter image description here" /></a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-20T18:43:43.353Z
Answers.Text: Image is from: https://spinningup.openai.com/en/latest/spinningup/rl_intro2.html
  
  Please credit sources.
Answers.UserDisplayName: Danyal
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-27T19:17:26.970Z
Answers.LastActivityDate: 2019-02-27T19:17:26.970Z
Answers.LastEditDate: 2020-06-17T09:57:20.710Z
Answers.UserDisplayName: rrz0
Answers.Body: <p>According to <em><a href="https://spinningup.openai.com/en/latest/spinningup/rl_intro2.html" rel="nofollow noreferrer">OpenAI – Kinds of RL Algorithms</a></em>, algorithms which use a model of the environment, i.e. a function which predicts state transitions and rewards, are called <strong>model-based</strong> methods, and those that don’t are called <strong>model-free</strong>. This model can either have been given the agent or learned by the agent.</p>
  
  <p>Using a model allows the agent to plan by thinking ahead, seeing what would happen for a range of possible choices, and explicitly deciding between its options. This may be useful when faced with problems that require more long-term thinking. One way to perform planning is by using some kind of tree search, for example <a href="https://en.wikipedia.org/wiki/Monte_Carlo_tree_search" rel="nofollow noreferrer">Monte Carlo tree search</a> (MCTS), or—which I suspect could also be used—<a href="https://en.wikipedia.org/wiki/Rapidly-exploring_random_tree#Variants_and_improvements_for_motion_planning" rel="nofollow noreferrer">variants</a> of the rapidly exploring random tree (RRT). See e.g. <em><a href="https://deepmind.com/blog/agents-imagine-and-plan/" rel="nofollow noreferrer">Agents that imagine and plan</a></em>.</p>
  
  <p>The agent can then distill the results from planning ahead into a learned policy – this is known as expert iteration.</p>
  
  <p>A model can also be used to create a simulated, or "imagined," environment in which the state is updated by using the model, and make the agent learn inside of that environment, such as in <em><a href="https://worldmodels.github.io/" rel="nofollow noreferrer">World Models</a></em>.</p>
  
  <p>In many real-world scenarios, the ground-truth model of the environment is not available to the agent. If an agent wants to use a model in this case, it has to learn the model, which can be challenging for several reasons.</p>
  
  <p>There are however cases in which the agent uses a model that is already known and consequently doesn't have to learn the model, such as in <a href="https://deepmind.com/blog/alphazero-shedding-new-light-grand-games-chess-shogi-and-go/" rel="nofollow noreferrer">AlphaZero</a>, where the model comes in form of the rules of the game.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-11T01:34:48.157Z
Answers.LastActivityDate: 2019-02-11T01:34:48.157Z
Answers.UserDisplayName: HelloGoodbye
Body: <p>What's the difference between model-free and model-based reinforcement learning? </p>
  
  <p>It seems to me that any model-free learner, learning through trial and error, could be reframed as model-based. In that case, when would model-free learners be appropriate?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-02-11T01:40:37.123Z
Comments.Text: How do you mean that you could reframe a model-free learner as a model-based?
Comments.UserDisplayName: HelloGoodbye
Tags: <reinforcement-learning><comparison><model-based-methods><model-free-methods>
Title: What's the difference between model-free and model-based reinforcement learning?
UserDisplayName: mynameisvinn
ThreadText: # What's the difference between model-free and model-based reinforcement learning?
  
  (Asked by: username_0 on 2017-11-07)
  
  What's the difference between model-free and model-based reinforcement learning?
  
  It seems to me that any model-free learner, learning through trial and error, could be reframed as model-based. In that case, when would model-free learners be appropriate?
  
  **Tags:** reinforcement-learning, comparison, model-based-methods, model-free-methods
  
  > **Comments:**
  > **username_1**: How do you mean that you could reframe a model-free learner as a model-based?  
  
  ---
  
  ## 6 answers
  
  ### Answer by username_2 on 2018-06-13. Score: 55
  
  > What's the difference between model-free and model-based reinforcement learning?
  
  In Reinforcement Learning, the terms "model-based" and "model-free" do *not* refer to the use of a neural network or other statistical learning model to predict values, or even to predict next state (although the latter may be used as part of a model-based algorithm and be called a "model" regardless of whether the algorithm is model-based or model-free).
  
  Instead, the term refers strictly as to whether, whilst during learning or acting, the agent uses predictions of the environment response. The agent can use a single prediction from the model of next reward and next state (a sample), or it can ask the model for the *expected* next reward, or the full *distribution* of next states and next rewards. These predictions can be provided entirely outside of the learning agent - e.g. by computer code that understands the rules of a dice or board game. Or they can be learned by the agent, in which case they will be approximate.
  
  Just because there is a model of the environment implemented, does not mean that a RL agent is "model-based". To qualify as "model-based", the learning algorithms have to explicitly reference the model:
  
  * Algorithms that purely sample from experience such as Monte Carlo Control, SARSA, Q-learning, Actor-Critic are "model free" RL algorithms. They rely on real samples from the environment and never use generated predictions of next state and next reward to alter behaviour (although they might sample from experience memory, which is close to being a model).
  * The archetypical model-based algorithms are Dynamic Programming (Policy Iteration and Value Iteration) - these all use the model's predictions or distributions of next state and reward in order to calculate optimal actions. Specifically in Dynamic Programming, the model must provide state transition probabilities, and expected reward from any state, action pair. Note this is rarely a learned model.
  * Basic TD learning, using state values only, must also be model-based in order to work as a control system and pick actions. In order to pick the best action, it needs to query a model that predicts what will happen on each action, and implement a policy like $\pi(s) = \text{argmax}\_a \sum\_{s',r} p(s',r|s,a)(r + v(s'))$ where $p(s',r|s,a)$ is the probability of receiving reward $r$ and next state $s'$ when taking action $a$ in state $s$. That function $p(s',r|s,a)$ is essentially the model.
  
  The RL literature differentiates between "model" as a model of the environment for "model-based" and "model-free" learning, and use of statistical learners, such as neural networks.
  
  In RL, neural networks are often employed to learn and generalise value functions, such as the Q value which predicts total return (sum of discounted rewards) given a state and action pair. Such a trained neural network is often called a "model" in e.g. supervised learning. However, in RL literature, you will see the term "function approximator" used for such a network to avoid ambiguity.
  
  > It seems to me that any model-free learner, learning through trial and error, could be reframed as model-based.
  
  I think here you are using the general understanding of the word "model" to include any structure that makes useful predictions. That would apply to e.g. table of Q values in SARSA.
  
  However, as explained above, that's not how the term is used in RL. So although your understanding that RL builds useful internal representations is correct, you are not technically correct that this can be used to re-frame between "model-free" as "model-based", because those terms have a very specific meaning in RL.
  
  > In that case, when would model-free learners be appropriate?
  
  Generally with current state of art in RL, if you don't have an accurate model provided as part of the problem definition, then model-free approaches are often superior.
  
  There is lots of interest in agents that build predictive models of the environment, and doing so as a "side effect" (whilst still being a model-free algorithm) can still be useful - it may regularise a neural network or help discover key predictive features that can also be used in policy or value networks. However, model-based agents that learn their own models for planning have a problem that inaccuracy in these models can cause instability (the inaccuracies multiply the further into the future the agent looks). Some promising inroads are being made using [imagination-based agents](https://www.deepmind.com/blog/agents-that-imagine-and-plan) and/or mechanisms for deciding when and how much to trust the learned model during planning.
  
  Right now (in 2018), if you have a real-world problem in an environment without an explicit known model at the start, then the safest bet is to use a model-free approach such as DQN or A3C. That may change as the field is moving fast and new more complex architectures could well be the norm in a few years.
  
  > **Comments:**
  > **username_3**: A small correction, normally the terms "model based" or "model free" are not used for planning algorithms such as MCTS. It is only used to classify learning algorithms.  
  > **username_2**: @username_3: I'm not 100% certain about that, but have removed the reference to MCTS. Out of interest, where would you place DynaQ regarding this limitation of the usage of the terms? I think it becomes tricky, when the algorithms all share such a common view of the MDP model and improving policies, to tell where the boundaris are between planning and learning.  
  > **username_3**: I could be wrong, I'm a novice in the area. I just remember a teacher from the field making that comment after I had done the same remark.  
  > **username_4**: @username_2, what would you recommend for an environment which is fully observable, but also gives back a model of it -- the exact reward and transition functions are known. I think there could be a benefit in using something beyond  DQN or A3C to take advantage of that knowledge. I cannot use dynamic programming since state-space is too large. thanks  
  > **username_2**: @username_4: I think that could be a new question on the site. In brief, you can still use most RL methods, and add *planning* to refine estimates of return values.  
  > **username_5**: Thank you for informative answer! I want to clarify one thing, for model-free algorithms you've said they "never use generated predictions of next state and next reward to alter behaviour" however in Q-learning agent may take greedy action(isn't this generated?) or random action even though it updates Q-value assuming agent took greedy action, isn't greedy action generated since it is not actually taken by the agent?  
  > **username_2**: @username_5 It is the prediction of next reward and state that are part of a model of the environment. The action is treated separately, using a policy of some kind, and yes all learning agents must have a method to do that somehow. The difference is important: An agent can control its action choice, but cannot control any other aspect of the environment rules. When looking at the difference between action taken (exploring behaviour policy) and the value update (greedy target policy), it doesn't matter whether you consider this "generated", it has nothing to do with the model of the environment  
  > **username_6**: @username_2 You say that "_if you have a real-world problem in an environment without an explicit known model at the start, then the safest bet is to use a model-free approach such as_ ". You mean that you should use real world simulations?  usually, when using a simulation even if the "learning agent" does not use the model, the model is necessary -and present-in order to excute the script and study the algorithm performance. So, apparently it is correct to say that in real world problem not only the "learning algorithm" is the problem.  
  > **username_6**: Continuation: You also need to construct a correct "real world simulation", except if you can experiment in the real world.  In this sense, RL is not model free. Am I wright?  
  > **username_2**: @username_6 There is a complex relationship between models, simulation and planning, in terms of when you might consider that you are using one or the other. One important detail is whether you have a sampling model or a distribution model. Model-free methods are often paired with simulations which are effectively sampling models. If the end goal is to then use the trained agent in a real environment, you might consider the training session as planning  
  > **username_7**: Other sources suggest TD-learning is model-free though.  
  > **username_2**: @username_7: Yes TD learning is itself model-free. It doesn't *require* any model. However, it is often used in situations where there is a model in the combined system. E.g. DynaQ.  
  > **username_7**: @username_2 so is the statement 'Basic TD learning, using state values only, must also be model-based in order to work as a control system and pick actions.' still correct (I was confused by it)?  
  > **username_2**: @username_7 - the learning system in that case must be model-based, yes. Without a model, TD learning using state values cannot make decisions. You cannot run value-based TD learning in a control scenario otehrwise, which is why you would typically use SARSA or Q learning (which are TD learning on action values) if you want a model-free TD learner. TD on state values still works model-free in predicion scenarios though.  
  
  ---
  
  ### Answer by username_8 on 2017-11-08. Score: 36
  
  Model-based reinforcement learning has an agent try to understand the world and create a model to represent it. Here the model is trying to capture 2 functions, the transition function from states $T$ and the reward function $R$. From this model, the agent has a reference and can plan accordingly.
  
  However, it is not necessary to learn a model, and the agent can instead learn a policy directly using algorithms like Q-learning or policy gradient.
  
  A simple check to see if an RL algorithm is model-based or model-free is:
  
  *If, after learning, the agent can make predictions about what the next state and reward will be before it takes each action, it's a model-based RL algorithm.*
  
  If it can't, then it’s a model-free algorithm.
  
  > **Comments:**
  > **username_9**: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/105284/discussion-on-answer-by-username_8-username_8-whats-the-difference-between-model-free-a).  
  
  ---
  
  ### Answer by username_9 on 2018-11-07. Score: 28
  
  In reinforcement learning (RL), there is an *agent* which interacts with an *environment* (in time steps). At each time step, the agent decides and executes an *action*, $a$, on an environment, and the environment responds to the agent by moving from the current *state* (of the environment), $s$, to the next state (of the environment), $s'$, and by emitting a scalar signal, called the *reward*, $r$. In principle, this interaction can continue forever or until e.g. the agent dies.
  
  The main goal of the agent is to collect the largest amount of reward "in the long run". To do that, the agent needs to find an optimal policy (roughly, the optimal strategy to behave in the environment). In general, a policy is a function which, given a current state of the environment, outputs an action (or a probability distribution over actions, if the policy is *stochastic*) to execute in the environment. A policy can thus be thought of as the "strategy" used by the agent to behave in this environment. An optimal policy (for a given environment) is a policy which, if followed, will make the agent collect the largest amount of reward in the long run (which is the goal of the agent). In RL, we are thus interested in finding optimal policies.
  
  The environment can be deterministic (that is, roughly, the same action in the same state leads to the same next state, for all time steps) or stochastic (or non-deterministic), that is, if the agent takes an action in a certain state, the resulting next state of the environment might not necessarily always be the same: there is a probability that it will be a certain state or another. Of course, these uncertainties will make the task of finding the optimal policy harder.
  
  In RL, the problem is often mathematically formulated as a [Markov decision process](https://en.wikipedia.org/wiki/Markov_decision_process) (MDP). A MDP is a way of representing the "dynamics" of the environment, that is, the way the environment will react to the possible actions the agent might take, at a given state. More precisely, an MDP is equipped with a **transition function** (or "transition model"), which is a function that, given the current state of the environment and an action (that the agent might take), outputs a probability of moving to any of the next states. A **reward function** is also associated with an MDP. Intuitively, the reward function outputs a reward, given the current state of the environment (and, possibly, an action taken by the agent and the next state of the environment). Collectively, the transition and reward functions are often called the **model** of environment. To conclude, the MDP is the problem and the solution to the problem is a policy. Furthermore, the "dynamics" of the environment are governed by the transition and reward functions (that is, the "model").
  
  However, we often do not have the MDP, that is, we do not have the transition and reward functions (of the MDP associated the environment). Hence, we cannot estimate a policy from the MDP, because it is unknown. Note that, in general, if we had the transition and reward functions of the MDP associated with the environment, we could exploit them and retrieve an optimal policy (using dynamic programming algorithms).
  
  In the absence of these functions (that is, when the MDP is unknown), to estimate the optimal policy, the agent needs to interact with environment and observe the responses of the environment. This is often referred to as the "reinforcement learning problem", because the agent will need to estimate a policy by *reinforcing* its beliefs about the dynamics of the environment. Over time, the agent starts to understand how the environment responds to its actions, and it can thus start to estimate the optimal policy. Thus, in the RL problem, the agent estimates the optimal policy to behave in an unknown (or partially known) environment by interacting with it (using a "trial-and-error" approach).
  
  In this context, a **model-based** algorithm is an algorithm that uses the transition function (and the reward function) in order to estimate the optimal policy. The agent might have access only to an approximation of the transition function and reward functions, which can be learned by the agent while it interacts with the environment or it can be given to the agent (e.g. by another agent). In general, in a model-based algorithm, the agent can potentially predict the dynamics of the environment (during or after the learning phase), because it has an estimate of the transition function (and reward function). However, note that the transition and reward functions that the agent uses in order to improve its estimate of the optimal policy might just be approximations of the "true" functions. Hence, the optimal policy might never be found (because of these approximations).
  
  A **model-free** algorithm is an algorithm that estimates the optimal policy without using or estimating the dynamics (transition and reward functions) of the environment. In practice, a model-free algorithm either estimates a "value function" or the "policy" directly from experience (that is, the interaction between the agent and environment), without using neither the transition function nor the reward function. A value function can be thought of as a function which evaluates a state (or an action taken in a state), for all states. From this value function, a policy can then be derived.
  
  In practice, one way to distinguish between model-based or model-free algorithms is to look at the algorithms and see if they use the transition or reward function.
  
  For instance, let's look at the main update rule in the *Q-learning algorithm*:
  
  $$Q(S\_t, A\_t) \leftarrow Q(S\_t, A\_t) + \alpha (R\_{t+1} + \gamma \max\_{a}Q(S\_{t+1}, a) - Q(S\_t, A\_t))$$
  
  As we can see, this update rule does not use any probabilities defined by the MDP. Note: $R\_{t+1}$ is just the reward that is obtained at the next time step (after taking the action), but it is not necessarily known beforehand. So, Q-learning is a model-free algorithm.
  
  Now, let's look at the main update rule of the *policy improvement* algorithm:
  
  $$Q(s,a) \leftarrow \sum\_{s' \in \mathcal{S}, r\in\mathcal{R}}p(s',r|s,a)(r+\gamma V(s'))$$
  
  We can immediately observe it uses $p(s',r|s,a)$, a probability defined by the MDP model. So, *policy iteration* (a dynamic programming algorithm), which uses the policy improvement algorithm, is a model-based algorithm.
  
  > **Comments:**
  > **username_10**: Already upvoted. Very well explained sir. Just a confusion, that why can't we just learn a reward and transition function associated with environment and then find optimal policy by dynamic learning. And if we are given this environment model by another agent then why can't we just apply dynamic learning to find optimal policy?  
  > **username_9**: @username_10 In principle, one can do what you are suggesting. However, as I say in my answer, if you first need to estimate the transition and reward functions, then you necessarily introduce more approximations in your calculations, which may not be desirable.  
  > **username_11**: When you are comparing the equations of update rules, I don't understand why the model-free still uses $R_{t+1}$. You explained that this is not necessarily known beforehand, but then how is it eventually known? What are the starting conditions of this reward if we don't have a model?  
  > **username_9**: @username_11 What I mean by "not known beforehand" is that $R_{t+1}$ is only known after you have taken the action $A_t$. $R_{t+1}$ is determined by the reward function, if $p(s', r \mid s, a)$ is not known. Note that the environment still behaves according to $p$ and the reward function, but $p$ may not be known.  
  
  ---
  
  ### Answer by username_3 on 2019-12-01. Score: 9
  
  Although there are several good answers, I want to add this paragraph from [Reinforcement Learning: An Introduction](http://incompleteideas.net/book/bookdraft2017nov5.pdf), page 303, for a more psychological view on the difference.
  
  > The distinction between model-free and model-based reinforcement learning algorithms
  > corresponds to the distinction psychologists make between habitual and goal-directed
  > control of learned behavioral patterns. Habits are behavior patterns triggered by appropriate
  > stimuli and then performed more-or-less automatically. Goal-directed behavior,
  > according to how psychologists use the phrase, is purposeful in the sense that it is controlled
  > by knowledge of the value of goals and the relationship between actions and their
  > consequences. Habits are sometimes said to be controlled by antecedent stimuli, whereas
  > goal-directed behavior is said to be controlled by its consequences (Dickinson, 1980,
  > 1985). Goal-directed control has the advantage that it can rapidly change an animal’s
  > behavior when the environment changes its way of reacting to the animal’s actions. While
  > habitual behavior responds quickly to input from an accustomed environment, it is unable
  > to quickly adjust to changes in the environment.
  
  It keeps going from there, and has a nice example afterwards.
  
  I think the main point that was not always explained in the other answers, is that in a model-free approach you still need some kind of environment to tell you what is the reward associated with your action. The big difference is that you do NOT need to store any information about the model. You give the environment your chosen action, you update your estimated policy, and you forget about it. On the other hand, in model-based approaches, you either need to know the state transitions history as in Dynamic Programming, or you need to be able to calculate all possible next states and associated rewards, from the present state.
  
  
  ---
  
  ### Answer by username_12 on 2019-02-27. Score: 5
  
  ## Model-Free RL
  
  In Model-Free RL, the agent does not have access to a model of the environment. By environment I mean a function which predicts state transition and rewards.
  
  As of the time of writing, model-free methods are more popular and have been researched extensively.
  
  ## Model-Based RL
  
  In Model-Based RL, the agent has access to a model of the environment.
  
  Main advantage is that this allows the agent to plan ahead by thinking ahead. Agents distill the results from planning ahead into a learned policy. A famous example of Model-Based RL is [AlphaZero](https://arxiv.org/abs/1712.01815).
  
  The main downside is that many times a ground-truth representation of the environment is not usually available.
  
  ---
  
  Below is a non-exhaustive taxonomy of RL algorithms, which may help you to visualize better the RL landscape.
  
  [![enter image description here](https://i.sstatic.net/oxKqm.png)](https://i.sstatic.net/oxKqm.png)
  
  > **Comments:**
  > **username_13**: Image is from: https://spinningup.openai.com/en/latest/spinningup/rl_intro2.html  Please credit sources.  
  
  ---
  
  ### Answer by username_1 on 2019-02-11. Score: 2
  
  According to *[OpenAI – Kinds of RL Algorithms](https://spinningup.openai.com/en/latest/spinningup/rl_intro2.html)*, algorithms which use a model of the environment, i.e. a function which predicts state transitions and rewards, are called **model-based** methods, and those that don’t are called **model-free**. This model can either have been given the agent or learned by the agent.
  
  Using a model allows the agent to plan by thinking ahead, seeing what would happen for a range of possible choices, and explicitly deciding between its options. This may be useful when faced with problems that require more long-term thinking. One way to perform planning is by using some kind of tree search, for example [Monte Carlo tree search](https://en.wikipedia.org/wiki/Monte_Carlo_tree_search) (MCTS), or—which I suspect could also be used—[variants](https://en.wikipedia.org/wiki/Rapidly-exploring_random_tree#Variants_and_improvements_for_motion_planning) of the rapidly exploring random tree (RRT). See e.g. *[Agents that imagine and plan](https://deepmind.com/blog/agents-imagine-and-plan/)*.
  
  The agent can then distill the results from planning ahead into a learned policy – this is known as expert iteration.
  
  A model can also be used to create a simulated, or "imagined," environment in which the state is updated by using the model, and make the agent learn inside of that environment, such as in *[World Models](https://worldmodels.github.io/)*.
  
  In many real-world scenarios, the ground-truth model of the environment is not available to the agent. If an agent wants to use a model in this case, it has to learn the model, which can be challenging for several reasons.
  
  There are however cases in which the agent uses a model that is already known and consequently doesn't have to learn the model, such as in [AlphaZero](https://deepmind.com/blog/alphazero-shedding-new-light-grand-games-chess-shogi-and-go/), where the model comes in form of the rules of the game.
  
  
  ---
  
  