Answers.Body: <p>I think that your description is roughly correct, but I wouldn't call a &quot;sampling model&quot; a &quot;model&quot; because it doesn't necessarily <a href="https://ai.stackexchange.com/a/25460/2444">model</a> something, unless, for example, you are first learning in simulation to later be able to act in the real-world or environment (in this sense, the simulation would be a model of the real environment, but this does not have to be the case, i.e. you may just want to act in the simulation (e.g. Atari games)), or, alternatively, when it's really a model of the MDP, but, in that case, you can just call it a model estimate.</p>
  <p>So, you can call it a</p>
  <ul>
  <li><strong>sampling function</strong>, in case you sample e.g. from the <a href="https://ai.stackexchange.com/q/6579/2444">experience replay</a></li>
  <li><strong>environment function</strong>, in case <span class="math-container">$r$</span> and <span class="math-container">$s'$</span> are returned by the environment,</li>
  <li><strong>model estimate</strong>, in case it's an estimate of <span class="math-container">$p(s' \mid s, a)$</span> (people may consider an experience replay a model estimate or, at least, information that can be used to build a model estimate)</li>
  </ul>
  <p>The important thing to keep in mind is that, if you want to take an action <span class="math-container">$a$</span> in a certain state <span class="math-container">$s$</span>, you need a function that returns you a reward <span class="math-container">$r$</span> and next state <span class="math-container">$s'$</span>, if you want to do reinforcement learning.</p>
  <p>I don't know if this distinction has been emphasized in the literature, but, as you noted, you can learn/estimate a (transition) model by exploring the world. I had asked a related question <a href="https://ai.stackexchange.com/q/10620/2444">here</a> a few years ago. You can also estimate the reward function, which is sometimes incorporated in the &quot;model&quot; of the environment, which, in this case, is denoted as <span class="math-container">$p(s', r \mid s, a)$</span> rather than just <span class="math-container">$p(s' \mid s, a)$</span>, but these terms can be written as a function of each other.</p>
  <p>People may also confuse this environment function with an <strong>exploratory policy</strong>, as they are, in a way, both used for exploration, but I think the concepts are distinct enough, as an exploratory policy is a way of deciding how to act given your current knowledge or ignorance: the exploratory policy can be viewed as a way of exercising/calling the environment function.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-08T11:52:59.083Z
Answers.Text: Thank you for the answer, @nbro. I also wouldn't a "sampling model" a "model",   but it seems to me that this concept falls within the definition of a "model" given by Sutton and Barto, which I quoted in the question.
  I appreciate your comments. That said, you didn't fully answer to questions 2 and 3, that were the core of the question :)
Answers.UserDisplayName: A. Pesare
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-08T12:37:03.313Z
Answers.Text: @A.Pesare Why does the "sampling model" fall into the category of "model"? A model in RL usually refers to a **probability distribution** that describes the dynamics of the environment, i.e. $p(s' \mid s, a)$. It only makes sense to call something a "sampling model" if that something is really a model or model estimate (as I call it) and if there's sampling involved (but you can also sample from $p(s' \mid s, a)$, so why not calling it also a sampling model?), if we want to be consistent with Sutton and Barto.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-08T12:40:36.477Z
Answers.Text: I've read many RL papers and I've never seen the word "sampling model" being used, as far as I remember. The closest to that is calling the experience replay technique a "model", but in its rawest form an ER is not a model, it's just a dataset, which you could use to build a model estimate. The word "sampling" is being used probably because we can "sample" from the "experience replay", which would be used to build the model, but this is a stretch. Again, the ER alone is not a model, it's a buffer/array/list/dataset of data, but it may induce a model.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-08T12:59:40.560Z
Answers.Text: If we follow the vague definition of a model that you quote "something that mimics the behavior of the environment, or more generally, that allows inferences to be made about how the environment will behave.", then yes you could consider an ER a model, but why call it a sampling model? As I said, yes, sometimes people may say that the ER is a model, but, in my view, ER is not directly a model, it's a dataset. It would be like calling a dataset in supervised learning a model too, but, in this case, it would be a different model. It doesn't make sense. The dataset is used to build the model.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-20T09:42:14.707Z
Answers.Text: By the way, [in chapter 8 (p. 159)](http://incompleteideas.net/book/RLbook2020.pdf#page=182) of Sutton & Barto's book, they also mention these _distribution model_ and _sampling model_.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-03T12:03:45.300Z
Answers.LastActivityDate: 2022-01-03T13:31:42.390Z
Answers.LastEditDate: 2022-01-03T13:31:42.390Z
Answers.UserDisplayName: nbro
Body: <p>The book from Sutton and Barto, <em>Reinforcement Learning: An Introduction</em>, define a <em>model</em> in Reinforcement Learning as</p>
  <blockquote>
  <p>something that mimics the behavior of the environment, or more generally, that allows inferences to be made about how the environment will behave.</p>
  </blockquote>
  <p>In <a href="https://ai.stackexchange.com/q/7628/2444">this answer</a>, the answerer makes a distinction:</p>
  <blockquote>
  <p>There are broadly two types of model:</p>
  <ul>
  <li><p>A <em>distribution model</em> which provides probabilities of all events. The most general function for this might be <span class="math-container">$p(r,s'|s,a)$</span> which is the probability of receiving reward <span class="math-container">$r$</span> and transitioning to state <span class="math-container">$s'$</span> given starting in state <span class="math-container">$s$</span> and taking action <span class="math-container">$a$</span>.</p>
  </li>
  <li><p>A <em>sampling model</em> which generates reward <span class="math-container">$r$</span> and next state <span class="math-container">$s'$</span> when given a current state <span class="math-container">$s$</span> and action <span class="math-container">$a$</span>. The samples might be from a simulation, or just taken from history of what the learning algorithm has experienced so far.</p>
  </li>
  </ul>
  </blockquote>
  <p>The main difference is that in sampling models I only have a black box, which, given a certain input <span class="math-container">$(s,a)$</span>, generates an output, but I don't know anything about the probability distributions of the MDP. However, having a sampling model, I can <em>reconstruct</em> (approximately) the probability distributions by running thousands of experiments (e.g. Monte Carlo Tree Search).</p>
  <p>On the other hand, if I have a distribution model, I can always sample from it.</p>
  <p>I was wondering if</p>
  <ol>
  <li><p>what I wrote is correct;</p>
  </li>
  <li><p>this distinction has been remarked in literature and where I can find a more in-depth discussion on the topic;</p>
  </li>
  <li><p>someone has ever separated <em>model-based</em> algorithms which use a distribution model and <em>model-based</em> algorithms which use only a sampling model.</p>
  </li>
  </ol>
  
Tags: <reinforcement-learning><comparison><reference-request><model-based-methods><transition-model>
Title: What is the difference between a distribution model and a sampling model in Reinforcement Learning?
UserDisplayName: A. Pesare
ThreadText: # What is the difference between a distribution model and a sampling model in Reinforcement Learning?
  
  (Asked by: username_0 on 2021-03-30)
  
  The book from Sutton and Barto, *Reinforcement Learning: An Introduction*, define a *model* in Reinforcement Learning as
  
  > something that mimics the behavior of the environment, or more generally, that allows inferences to be made about how the environment will behave.
  
  In [this answer](https://ai.stackexchange.com/q/7628/2444), the answerer makes a distinction:
  
  > There are broadly two types of model:
  >
  > * A *distribution model* which provides probabilities of all events. The most general function for this might be $p(r,s'|s,a)$ which is the probability of receiving reward $r$ and transitioning to state $s'$ given starting in state $s$ and taking action $a$.
  > * A *sampling model* which generates reward $r$ and next state $s'$ when given a current state $s$ and action $a$. The samples might be from a simulation, or just taken from history of what the learning algorithm has experienced so far.
  
  The main difference is that in sampling models I only have a black box, which, given a certain input $(s,a)$, generates an output, but I don't know anything about the probability distributions of the MDP. However, having a sampling model, I can *reconstruct* (approximately) the probability distributions by running thousands of experiments (e.g. Monte Carlo Tree Search).
  
  On the other hand, if I have a distribution model, I can always sample from it.
  
  I was wondering if
  
  1. what I wrote is correct;
  2. this distinction has been remarked in literature and where I can find a more in-depth discussion on the topic;
  3. someone has ever separated *model-based* algorithms which use a distribution model and *model-based* algorithms which use only a sampling model.
  
  **Tags:** reinforcement-learning, comparison, reference-request, model-based-methods, transition-model
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-01-03. Score: 1
  
  I think that your description is roughly correct, but I wouldn't call a "sampling model" a "model" because it doesn't necessarily [model](https://ai.stackexchange.com/a/25460/2444) something, unless, for example, you are first learning in simulation to later be able to act in the real-world or environment (in this sense, the simulation would be a model of the real environment, but this does not have to be the case, i.e. you may just want to act in the simulation (e.g. Atari games)), or, alternatively, when it's really a model of the MDP, but, in that case, you can just call it a model estimate.
  
  So, you can call it a
  
  * **sampling function**, in case you sample e.g. from the [experience replay](https://ai.stackexchange.com/q/6579/2444)
  * **environment function**, in case $r$ and $s'$ are returned by the environment,
  * **model estimate**, in case it's an estimate of $p(s' \mid s, a)$ (people may consider an experience replay a model estimate or, at least, information that can be used to build a model estimate)
  
  The important thing to keep in mind is that, if you want to take an action $a$ in a certain state $s$, you need a function that returns you a reward $r$ and next state $s'$, if you want to do reinforcement learning.
  
  I don't know if this distinction has been emphasized in the literature, but, as you noted, you can learn/estimate a (transition) model by exploring the world. I had asked a related question [here](https://ai.stackexchange.com/q/10620/2444) a few years ago. You can also estimate the reward function, which is sometimes incorporated in the "model" of the environment, which, in this case, is denoted as $p(s', r \mid s, a)$ rather than just $p(s' \mid s, a)$, but these terms can be written as a function of each other.
  
  People may also confuse this environment function with an **exploratory policy**, as they are, in a way, both used for exploration, but I think the concepts are distinct enough, as an exploratory policy is a way of deciding how to act given your current knowledge or ignorance: the exploratory policy can be viewed as a way of exercising/calling the environment function.
  
  > **Comments:**
  > **username_0**: Thank you for the answer, @nbro. I also wouldn't a "sampling model" a "model",   but it seems to me that this concept falls within the definition of a "model" given by Sutton and Barto, which I quoted in the question. I appreciate your comments. That said, you didn't fully answer to questions 2 and 3, that were the core of the question :)  
  > **username_1**: @A.Pesare Why does the "sampling model" fall into the category of "model"? A model in RL usually refers to a **probability distribution** that describes the dynamics of the environment, i.e. $p(s' \mid s, a)$. It only makes sense to call something a "sampling model" if that something is really a model or model estimate (as I call it) and if there's sampling involved (but you can also sample from $p(s' \mid s, a)$, so why not calling it also a sampling model?), if we want to be consistent with Sutton and Barto.  
  > **username_1**: I've read many RL papers and I've never seen the word "sampling model" being used, as far as I remember. The closest to that is calling the experience replay technique a "model", but in its rawest form an ER is not a model, it's just a dataset, which you could use to build a model estimate. The word "sampling" is being used probably because we can "sample" from the "experience replay", which would be used to build the model, but this is a stretch. Again, the ER alone is not a model, it's a buffer/array/list/dataset of data, but it may induce a model.  
  > **username_1**: If we follow the vague definition of a model that you quote "something that mimics the behavior of the environment, or more generally, that allows inferences to be made about how the environment will behave.", then yes you could consider an ER a model, but why call it a sampling model? As I said, yes, sometimes people may say that the ER is a model, but, in my view, ER is not directly a model, it's a dataset. It would be like calling a dataset in supervised learning a model too, but, in this case, it would be a different model. It doesn't make sense. The dataset is used to build the model.  
  > **username_1**: By the way, [in chapter 8 (p. 159)](http://incompleteideas.net/book/RLbook2020.pdf#page=182) of Sutton & Barto's book, they also mention these _distribution model_ and _sampling model_.  
  
  ---
  
  