Answers.Body: <p><strong>Transition Probabilities:</strong> Consider that you are at state <span class="math-container">$s$</span> and from that state take an action <span class="math-container">$a$</span>.Then there are some probability you will land up at state <span class="math-container">$s_{1}'$</span> or <span class="math-container">$s_{2}'$</span> (<span class="math-container">$s'$</span> indicate the next states). Those probabilities are called transition probabilities. In this example, the transition matrix is just a 3D array since it depends on your state and action(<span class="math-container">$p(s, a)$</span>).</p>
  
  <p><strong>Action value function <span class="math-container">$Q_{\pi}(s, a)$</span>:</strong> It is the expected total reward you get from state <span class="math-container">$s$</span>, taking action <span class="math-container">$a$</span> and thereafter following the policy, <span class="math-container">$\pi$</span>. </p>
  
  <blockquote>
    <p>Is the state transition in MDP stochastic transition, meaning transition to some other state without taking any action?</p>
  </blockquote>
  
  <p>The environment can be stochastic or deterministic. If the environment is stochastic then those transitions are stochastic. If the environment is deterministic then those transitions are deterministic.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-28T16:35:23.243Z
Answers.Text: Yes I am aware of transition probabilities, but what I am asking is how does that transition occur? Is the transition occur by taking "action" or is a stochastic phenomena? In other words how is the transition matrix populated? In other words the transition probability matrix arises if we record action over large period of time or its just that if you in state 1 there is simply stocastic chance that you will end up in state say 7. How do you end up in state 7 from state 1? by taking an action?
Answers.UserDisplayName: gfdsal
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-28T17:01:06.247Z
Answers.Text: The transition occurs by taking action. For example, if you take action 'left' then there is some chance you land up in the left state and there is also some probability that you will land up in other states than the left state. This thing will happen if the environment is stochastic.  In reinforcement learning, if you use dynamic programming, then you have to know the dynamics of the system(means the transition matrix and rewards). Or another thing you can do is go to the environment collect a lot of samples and calculate transition matrix from your collected samples..
Answers.UserDisplayName: Swakshar Deb
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-28T17:19:07.473Z
Answers.Text: ok so in MDF we build the transition matrix by taking actions and recording their frequency?
Answers.UserDisplayName: gfdsal
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-28T17:22:06.873Z
Answers.Text: yes, we can find the transition matrix by using those samples(this is called model-based reinforcement learning).
Answers.UserDisplayName: Swakshar Deb
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-28T17:49:14.053Z
Answers.Text: ok makes sense.
Answers.UserDisplayName: gfdsal
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-28T15:48:46.520Z
Answers.LastActivityDate: 2020-04-28T15:48:46.520Z
Answers.UserDisplayName: Swakshar Deb
Body: <p>Let's say we have MDP where we have a state transition matrix. </p>
  
  <p>How is this state transition different from action value in reinforcement learning? Is the state transition in MDP stochastic transition, meaning transition to some other state without taking any action?</p>
  
Tags: <reinforcement-learning><comparison><markov-decision-process><value-functions>
Title: What is the difference between the state transition of an MDP and an action-value?
UserDisplayName: gfdsal
ThreadText: # What is the difference between the state transition of an MDP and an action-value?
  
  (Asked by: username_0 on 2020-04-28)
  
  Let's say we have MDP where we have a state transition matrix.
  
  How is this state transition different from action value in reinforcement learning? Is the state transition in MDP stochastic transition, meaning transition to some other state without taking any action?
  
  **Tags:** reinforcement-learning, comparison, markov-decision-process, value-functions
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-04-28. Score: 0 | Accepted answer
  
  **Transition Probabilities:** Consider that you are at state $s$ and from that state take an action $a$.Then there are some probability you will land up at state $s\_{1}'$ or $s\_{2}'$ ($s'$ indicate the next states). Those probabilities are called transition probabilities. In this example, the transition matrix is just a 3D array since it depends on your state and action($p(s, a)$).
  
  **Action value function $Q\_{\pi}(s, a)$:** It is the expected total reward you get from state $s$, taking action $a$ and thereafter following the policy, $\pi$.
  
  > Is the state transition in MDP stochastic transition, meaning transition to some other state without taking any action?
  
  The environment can be stochastic or deterministic. If the environment is stochastic then those transitions are stochastic. If the environment is deterministic then those transitions are deterministic.
  
  > **Comments:**
  > **username_0**: Yes I am aware of transition probabilities, but what I am asking is how does that transition occur? Is the transition occur by taking "action" or is a stochastic phenomena? In other words how is the transition matrix populated? In other words the transition probability matrix arises if we record action over large period of time or its just that if you in state 1 there is simply stocastic chance that you will end up in state say 7. How do you end up in state 7 from state 1? by taking an action?  
  > **username_1**: The transition occurs by taking action. For example, if you take action 'left' then there is some chance you land up in the left state and there is also some probability that you will land up in other states than the left state. This thing will happen if the environment is stochastic.  In reinforcement learning, if you use dynamic programming, then you have to know the dynamics of the system(means the transition matrix and rewards). Or another thing you can do is go to the environment collect a lot of samples and calculate transition matrix from your collected samples..  
  > **username_0**: ok so in MDF we build the transition matrix by taking actions and recording their frequency?  
  > **username_1**: yes, we can find the transition matrix by using those samples(this is called model-based reinforcement learning).  
  > **username_0**: ok makes sense.  
  
  ---
  
  