Answers.Body: <blockquote>
    <p>If a policy is fixed, it is said that an MDP becomes an MRP.</p>
  </blockquote>
  
  <p>I would change the phrasing slightly here, to:</p>
  
  <p><em>If a policy is fixed, an MDP can be accurately modeled as an MRP.</em></p>
  
  <blockquote>
    <p>Why is this so? Aren't the transitions and rewards still parameterized by the action and current state? In other words, aren't the transition and reward matrices still cubes?</p>
  </blockquote>
  
  <p>The transition and reward matrices remain the same in the MDP, but it is <em>possible</em> to flatten them into an equivalent MRP, because in terms of observations of next state and reward, the action that is taken is just part of the transition rules - if the policy is fixed, then so are all the probabilities for next state and reward.</p>
  
  <p>More concretely, if you have an MDP with <span class="math-container">$|\mathcal{A}|$</span> transition matrices <span class="math-container">$P_{ss'}^a$</span> and a fixed policy <span class="math-container">$\pi(a|s)$</span>, then you can create a <em>combined</em> transition matrix with a sum:</p>
  
  <p><span class="math-container">$$P_{ss'} = \sum_{a \in \mathcal{A}} \pi(a|s) P_{ss'}^a$$</span> </p>
  
  <p>and you can similarly reduce the reward function. Once you have done so you have data that describe an MRP.</p>
  
  <blockquote>
    <p>How is it that it switches to an MRP which is not affected by actions?</p>
  </blockquote>
  
  <p>If the MDP represents a real system where actions are still being taken by an agent, then of course those are still present within the system, and still affect it. The difference is that if you know the agent's policy, then the action choice is predictable, and the MRP representation covers the full definition of probabilities of observed state transitions and rewards.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-30T06:56:53.027Z
Answers.LastActivityDate: 2019-09-30T08:42:12.170Z
Answers.LastEditDate: 2019-09-30T08:42:12.170Z
Answers.UserDisplayName: Neil Slater
Body: <p>If a policy is fixed, it is said that a Markov Decision Process (MDP) becomes a Markov Reward Process (MRP).</p>
  <p>Why is this so? Aren't the transitions and rewards still parameterized by the action and current state? In other words, aren't the transition and reward matrices still cubes?</p>
  <p>From my current train of thought, the only thing that is different is that the policy is not changing (the agent is not learning the policy). Everything else is the same.</p>
  <p>How is it that it switches to an MRP, which is not affected by actions?</p>
  <p>I am reading &quot;Deep Reinforcement Learning Hands-On&quot; by Maxim Lapan, which states this. I have also found this statement in online articles, but I cannot seem to wrap my head around it.</p>
  
Tags: <reinforcement-learning><markov-decision-process><policies><markov-reward-process>
Title: Why does having a fixed policy change a Markov Decision Process to a Markov Reward Process?
UserDisplayName: Peter
ThreadText: # Why does having a fixed policy change a Markov Decision Process to a Markov Reward Process?
  
  (Asked by: username_0 on 2019-09-30)
  
  If a policy is fixed, it is said that a Markov Decision Process (MDP) becomes a Markov Reward Process (MRP).
  
  Why is this so? Aren't the transitions and rewards still parameterized by the action and current state? In other words, aren't the transition and reward matrices still cubes?
  
  From my current train of thought, the only thing that is different is that the policy is not changing (the agent is not learning the policy). Everything else is the same.
  
  How is it that it switches to an MRP, which is not affected by actions?
  
  I am reading "Deep Reinforcement Learning Hands-On" by Maxim Lapan, which states this. I have also found this statement in online articles, but I cannot seem to wrap my head around it.
  
  **Tags:** reinforcement-learning, markov-decision-process, policies, markov-reward-process
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-09-30. Score: 1 | Accepted answer
  
  > If a policy is fixed, it is said that an MDP becomes an MRP.
  
  I would change the phrasing slightly here, to:
  
  *If a policy is fixed, an MDP can be accurately modeled as an MRP.*
  
  > Why is this so? Aren't the transitions and rewards still parameterized by the action and current state? In other words, aren't the transition and reward matrices still cubes?
  
  The transition and reward matrices remain the same in the MDP, but it is *possible* to flatten them into an equivalent MRP, because in terms of observations of next state and reward, the action that is taken is just part of the transition rules - if the policy is fixed, then so are all the probabilities for next state and reward.
  
  More concretely, if you have an MDP with $|\mathcal{A}|$ transition matrices $P\_{ss'}^a$ and a fixed policy $\pi(a|s)$, then you can create a *combined* transition matrix with a sum:
  
  $$P\_{ss'} = \sum\_{a \in \mathcal{A}} \pi(a|s) P\_{ss'}^a$$
  
  and you can similarly reduce the reward function. Once you have done so you have data that describe an MRP.
  
  > How is it that it switches to an MRP which is not affected by actions?
  
  If the MDP represents a real system where actions are still being taken by an agent, then of course those are still present within the system, and still affect it. The difference is that if you know the agent's policy, then the action choice is predictable, and the MRP representation covers the full definition of probabilities of observed state transitions and rewards.
  
  
  ---
  
  