Answers.Body: <p>Your understanding is right!</p>
  
  <p>Using a probabilistic transition function allows the model to explore a bigger search space before making a decision. One of the most important use cases of MDP is in POS tagging in NLP using a Hidden Markov Model.</p>
  
  <p>In case of a deterministic model, search space is limited by the number of transitions and hence at each step, a definite decision is made. This does not take into regard the possibility of relationship between the previous states, rather only deals with the current and next state. These model are good for solving a certain range of tasks like decision trees etc.</p>
  
  <p>When it come to tasks such as weather prediction, there is significance to the historical weather data. In such cases, we cannot make use of a deterministic approach. You always predict the <code>chance</code> of rainfall etc.</p>
  
  <p><a href="https://i.sstatic.net/QJclo.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/QJclo.jpg" alt="Weather prediction"></a></p>
  
  <p>This example can also be extended to predict the weather for future days</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-17T17:58:45.610Z
Answers.Text: To clarify your answer, I believe you have confirmed my understanding and then contributed two more things. First you pointed out that we want use the MDP to explore different options and solutions, so the probabilistic model enables this. Secondly you gave an example about weather, in which the fact that we don't store historical weather data is why our state is insufficient to allow our MDP state transition to be deterministic. Is this correct?
Answers.UserDisplayName: personjerry
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-20T11:32:09.297Z
Answers.Text: @personjerry the second part of your question that "we dont store historical weather data is why our state is insufficient to allow our MDP state transition to be deterministic", this is not correct. Even if you have recorded all the historical data, your weather prediction will still be indeterministic hence we make use of probability/chance. Your initial understanding that the world is deterministic is wrong, we do have purely stochastic systems
Answers.UserDisplayName: user0193
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-17T10:38:53.270Z
Answers.LastActivityDate: 2019-06-17T10:38:53.270Z
Answers.UserDisplayName: skillsmuggler
Body: <p>I've read that for MDPs the state transition function <span class="math-container">$P_a(s, s')$</span> is a probability. This seems strange to me for modeling because most environments (like video games) are deterministic.</p>
  
  <p>Now, I'd like to assert that most systems we work with are deterministic given enough information in the state (i.e. in a video game, if you had the random number seed, you could predict 'rolls', and then everything else follows game logic).</p>
  
  <p>So, my guess for why would MDP state transitions are probabilities is because the state given to the MDP is typically a subset (i.e. from feature engineering) of total information available. That, and of course to model non-deterministic systems.</p>
  
  <p>Is my understanding correct?</p>
  
Tags: <machine-learning><reinforcement-learning><markov-decision-process>
Title: Why are state transitions in MDPs probabilistic rather than deterministic?
UserDisplayName: personjerry
ThreadText: # Why are state transitions in MDPs probabilistic rather than deterministic?
  
  (Asked by: username_0 on 2019-06-17)
  
  I've read that for MDPs the state transition function $P\_a(s, s')$ is a probability. This seems strange to me for modeling because most environments (like video games) are deterministic.
  
  Now, I'd like to assert that most systems we work with are deterministic given enough information in the state (i.e. in a video game, if you had the random number seed, you could predict 'rolls', and then everything else follows game logic).
  
  So, my guess for why would MDP state transitions are probabilities is because the state given to the MDP is typically a subset (i.e. from feature engineering) of total information available. That, and of course to model non-deterministic systems.
  
  Is my understanding correct?
  
  **Tags:** machine-learning, reinforcement-learning, markov-decision-process
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-06-17. Score: 2 | Accepted answer
  
  Your understanding is right!
  
  Using a probabilistic transition function allows the model to explore a bigger search space before making a decision. One of the most important use cases of MDP is in POS tagging in NLP using a Hidden Markov Model.
  
  In case of a deterministic model, search space is limited by the number of transitions and hence at each step, a definite decision is made. This does not take into regard the possibility of relationship between the previous states, rather only deals with the current and next state. These model are good for solving a certain range of tasks like decision trees etc.
  
  When it come to tasks such as weather prediction, there is significance to the historical weather data. In such cases, we cannot make use of a deterministic approach. You always predict the `chance` of rainfall etc.
  
  [![Weather prediction](https://i.sstatic.net/QJclo.jpg)](https://i.sstatic.net/QJclo.jpg)
  
  This example can also be extended to predict the weather for future days
  
  > **Comments:**
  > **username_0**: To clarify your answer, I believe you have confirmed my understanding and then contributed two more things. First you pointed out that we want use the MDP to explore different options and solutions, so the probabilistic model enables this. Secondly you gave an example about weather, in which the fact that we don't store historical weather data is why our state is insufficient to allow our MDP state transition to be deterministic. Is this correct?  
  > **username_2**: @username_0 the second part of your question that "we dont store historical weather data is why our state is insufficient to allow our MDP state transition to be deterministic", this is not correct. Even if you have recorded all the historical data, your weather prediction will still be indeterministic hence we make use of probability/chance. Your initial understanding that the world is deterministic is wrong, we do have purely stochastic systems  
  
  ---
  
  