Answers.Body: <p>The MDP defines the environment (which corresponds to the task that you need to solve), so it defines e.g. the states of the environment, the actions that you <em>can</em> take in those states, the probabilities of transitioning from one state to the other and the probabilities of getting a reward when you take a certain action in a certain state. </p>
  
  <p>The policy corresponds to a strategy that the RL agent can follow to act in that environment. Note that the MDP doesn't define what the agent does in each state. That's why you need the policy! An optimal policy for a specific MDP corresponds to the strategy that, if followed, is guaranteed to give you the highest amount of reward in that environment. However, there are multiple strategies, most of them are not optimal. This should clarify why the policy is not part of the definition of the MDP.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-23T20:24:16.827Z
Answers.LastActivityDate: 2020-05-23T20:24:16.827Z
Answers.UserDisplayName: nbro
Answers.Body: <p>Aside from the points raised in <a href="https://ai.stackexchange.com/a/21437/1641">nbro's answer</a>, I'd like to point out that for a single MDP (a single instance of a "problem"), it may be sensible to study it from perspectives that include no policy at all, or multiple different policies.</p>
  
  <p>For instance, if I have an MDP, I may be interested in studying it by looking at various inherent properties of the environment. And if I then have multiple different MDPs, all without any policies or anything like that, I could compare them based on those properties. For example, I might simply want to measure the sizes of the state and action spaces. Or write out something like a game tree, and measure properties like the branching factor and the average / min / max / median depth at which we can find a terminal state.</p>
  
  <p>On the other hand, it can also be interesting sometimes to study multiple different policies all for the same MDP. A very common example would be any off-policy learning algorithm (like <span class="math-container">$Q$</span>-learning): they all involve at least one "target policy" (for which they're learning the <span class="math-container">$Q(s, a)$</span> values -- usually the greedy policy with respect to the values learned so far), and at least one "behaviour policy" (which they're using to generate experience -- often something like an <span class="math-container">$\epsilon$</span>-greedy policy). A more complex example would be population-based training setups, like the one <a href="https://doi.org/10.1038/s41586-019-1724-z" rel="nofollow noreferrer">DeepMind used for their StarCraft 2 training</a>; here they have a large population of different policies that they're all using in a complex training setup (and technically I suppose we should say they also have many different MDPs, where every combination of StarCraft 2 level + training opponent would formally be a different MDP).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-24T10:24:13.213Z
Answers.LastActivityDate: 2020-05-24T10:24:13.213Z
Answers.UserDisplayName: Dennis Soemers
Answers.Body: <p>As already explained by others, the policy accounts for the agent's decisions, which are not set by the enviornment.</p>
  <p>The only requirement of an MDP is to define a <em>space of possible policies</em> from which the agent can sample from (the &quot;D&quot; of an MDP).
  This is usually skipped in literature as it is assumed that the agent can sample <em>any</em> policy, even if this is true only under strong conditions.</p>
  <p>Instead, a Markov Reward Process (MRP) is an MDP with a fixed policy <span class="math-container">$\pi$</span>, and so is a Markov Process (MP), which is an MRP without rewards.</p>
  <p>I think David Silver's slides explain this very plainly, you might find it useful:
  <a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/MDP.pdf" rel="nofollow noreferrer">https://www.davidsilver.uk/wp-content/uploads/2020/03/MDP.pdf</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-01T10:13:46.573Z
Answers.LastActivityDate: 2023-08-01T10:13:46.573Z
Answers.UserDisplayName: Eduardo Pignatelli
Body: <p>I'm reading an article on reinforcement learning, and I don't understand why the agent's policy <span class="math-container">$\pi$</span> is not part of definition of Markov Decision process(MDP):</p>
  <blockquote>
  <p><a href="https://i.sstatic.net/giJ04.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/giJ04.png" alt="enter image description here" /></a></p>
  <p>Bu, Lucian, Robert Babu, and Bart De Schutter. &quot;A comprehensive survey of multiagent reinforcement learning.&quot; IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) 38.2 (2008): 156-172.</p>
  </blockquote>
  <p>My question is:</p>
  <blockquote>
  <p>Why the policy is not a part of the MDP definition?</p>
  </blockquote>
  
Tags: <reinforcement-learning><definitions><markov-decision-process><policies>
Title: Why is the policy not a part of the MDP definition?
UserDisplayName: Raphael Augusto
ThreadText: # Why is the policy not a part of the MDP definition?
  
  (Asked by: username_0 on 2020-05-23)
  
  I'm reading an article on reinforcement learning, and I don't understand why the agent's policy $\pi$ is not part of definition of Markov Decision process(MDP):
  
  > [![enter image description here](https://i.sstatic.net/giJ04.png)](https://i.sstatic.net/giJ04.png)
  >
  > Bu, Lucian, Robert Babu, and Bart De Schutter. "A comprehensive survey of multiagent reinforcement learning." IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) 38.2 (2008): 156-172.
  
  My question is:
  
  > Why the policy is not a part of the MDP definition?
  
  **Tags:** reinforcement-learning, definitions, markov-decision-process, policies
  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_1 on 2020-05-23. Score: 7
  
  The MDP defines the environment (which corresponds to the task that you need to solve), so it defines e.g. the states of the environment, the actions that you *can* take in those states, the probabilities of transitioning from one state to the other and the probabilities of getting a reward when you take a certain action in a certain state.
  
  The policy corresponds to a strategy that the RL agent can follow to act in that environment. Note that the MDP doesn't define what the agent does in each state. That's why you need the policy! An optimal policy for a specific MDP corresponds to the strategy that, if followed, is guaranteed to give you the highest amount of reward in that environment. However, there are multiple strategies, most of them are not optimal. This should clarify why the policy is not part of the definition of the MDP.
  
  
  ---
  
  ### Answer by username_2 on 2020-05-24. Score: 2
  
  Aside from the points raised in [username_1's answer](https://ai.stackexchange.com/a/21437/1641), I'd like to point out that for a single MDP (a single instance of a "problem"), it may be sensible to study it from perspectives that include no policy at all, or multiple different policies.
  
  For instance, if I have an MDP, I may be interested in studying it by looking at various inherent properties of the environment. And if I then have multiple different MDPs, all without any policies or anything like that, I could compare them based on those properties. For example, I might simply want to measure the sizes of the state and action spaces. Or write out something like a game tree, and measure properties like the branching factor and the average / min / max / median depth at which we can find a terminal state.
  
  On the other hand, it can also be interesting sometimes to study multiple different policies all for the same MDP. A very common example would be any off-policy learning algorithm (like $Q$-learning): they all involve at least one "target policy" (for which they're learning the $Q(s, a)$ values -- usually the greedy policy with respect to the values learned so far), and at least one "behaviour policy" (which they're using to generate experience -- often something like an $\epsilon$-greedy policy). A more complex example would be population-based training setups, like the one [DeepMind used for their StarCraft 2 training](https://doi.org/10.1038/s41586-019-1724-z); here they have a large population of different policies that they're all using in a complex training setup (and technically I suppose we should say they also have many different MDPs, where every combination of StarCraft 2 level + training opponent would formally be a different MDP).
  
  
  ---
  
  ### Answer by username_3 on 2023-08-01. Score: 1
  
  As already explained by others, the policy accounts for the agent's decisions, which are not set by the enviornment.
  
  The only requirement of an MDP is to define a *space of possible policies* from which the agent can sample from (the "D" of an MDP).
  This is usually skipped in literature as it is assumed that the agent can sample *any* policy, even if this is true only under strong conditions.
  
  Instead, a Markov Reward Process (MRP) is an MDP with a fixed policy $\pi$, and so is a Markov Process (MP), which is an MRP without rewards.
  
  I think David Silver's slides explain this very plainly, you might find it useful:
  <https://www.davidsilver.uk/wp-content/uploads/2020/03/MDP.pdf>
  
  
  ---
  
  