Answers.Body: <p>Yes, in which case, it will be more like a search problem, if it is not POMDP and have finite number of states. Or you can use the same framework (used for POMDP) with constrained (deterministic) transition matrix modeling for model-based systems.</p>
  <p>If you think about it after you train any model/agent with an MDP modeling, during test time the optimal strategy is generally deterministic, i.e., given a feature/state you will take a particular action even if it has more than one non-zero element in each row of the transition matrix.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-26T14:36:24.907Z
Answers.Text: Alright, when you say deterministic strategy you mean we select argmax or the maximum result? Can you please elaborate constrained (deterministic) transition matrix
Answers.UserDisplayName: gfdsal
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-28T00:37:28.610Z
Answers.Text: 1. Yes, I was referring to the best strategy obtained with argmax.
Answers.UserDisplayName: saiRegrefree
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-28T00:44:53.107Z
Answers.Text: 2. When you have a deterministic transition, the actions are irrelevant, since choosing an action is equivalent to reaching a state, so you can in simple cases use dynamic programing/Beam search to solve the problem. In complex cases, the reward can be infinite and you can use value iteration to solve the problem. I found some paper on the topic which might help you related work "Max-Plus Matching Pursuit for Deterministic Markov
  Decision Processes" https://arxiv.org/pdf/1906.08524.pdf
Answers.UserDisplayName: saiRegrefree
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-28T12:19:39.753Z
Answers.Text: allright makes sense! I accepted your answer as now it is clear. I have a small followup question. How can a transition be deterministic if the environment is stochastic? So say if I move left there is 30% chance that i will transition to front instead of left. So when you say actions are irrelevant, you mean to say we dont care about the 30% chance of transitioning to front if we intend to move left?
Answers.UserDisplayName: gfdsal
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-01T00:25:24.577Z
Answers.Text: I think there is a misunderstanding. Transition matrix being deterministic means when you take an action you are certain that you will end up in only one state, which means the environment is deterministic. In this case (deterministic environment/transition matrix) actions are irrelevant. "So say if I move left there is 30% chance that i will transition to front instead of left." - this means the environment/transition matrix is stochastic.
Answers.UserDisplayName: saiRegrefree
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-01T12:14:47.073Z
Answers.Text: +1 This clarifies my confusion. I was confused by the underlying assumption where environment was stochastic and transition matrix was deterministic.
Answers.UserDisplayName: gfdsal
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-25T18:34:36.780Z
Answers.LastActivityDate: 2020-06-25T18:34:36.780Z
Answers.UserDisplayName: saiRegrefree
Answers.Body: <p>Yes , a transition model shows that our environment is stochastic in nature, and with that model we know the probability of entering a state when an action is taken.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-26T22:24:15.380Z
Answers.LastActivityDate: 2020-06-26T22:24:15.380Z
Answers.UserDisplayName: Chukwudi
Body: <p>I am looking at a lecture on <a href="https://youtu.be/I2uSCTUHsUI?t=3951" rel="nofollow noreferrer">POMDP</a>, and the context is that, when the quadcopter can't see the landmarks, it has to use reckoning. And then he mentions the transition model is not deterministic, hence the uncertainty grows.</p>
  <p>Can transition models in MDP be deterministic?</p>
  
Tags: <reinforcement-learning><markov-decision-process><pomdp><transition-model>
Title: Does "transition model" alone in an MDP imply it's non-deterministic?
UserDisplayName: gfdsal
ThreadText: # Does "transition model" alone in an MDP imply it's non-deterministic?
  
  (Asked by: username_0 on 2020-06-25)
  
  I am looking at a lecture on [POMDP](https://youtu.be/I2uSCTUHsUI?t=3951), and the context is that, when the quadcopter can't see the landmarks, it has to use reckoning. And then he mentions the transition model is not deterministic, hence the uncertainty grows.
  
  Can transition models in MDP be deterministic?
  
  **Tags:** reinforcement-learning, markov-decision-process, pomdp, transition-model
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2020-06-25. Score: 0 | Accepted answer
  
  Yes, in which case, it will be more like a search problem, if it is not POMDP and have finite number of states. Or you can use the same framework (used for POMDP) with constrained (deterministic) transition matrix modeling for model-based systems.
  
  If you think about it after you train any model/agent with an MDP modeling, during test time the optimal strategy is generally deterministic, i.e., given a feature/state you will take a particular action even if it has more than one non-zero element in each row of the transition matrix.
  
  > **Comments:**
  > **username_0**: Alright, when you say deterministic strategy you mean we select argmax or the maximum result? Can you please elaborate constrained (deterministic) transition matrix  
  > **username_1**: 1. Yes, I was referring to the best strategy obtained with argmax.  
  > **username_1**: 2. When you have a deterministic transition, the actions are irrelevant, since choosing an action is equivalent to reaching a state, so you can in simple cases use dynamic programing/Beam search to solve the problem. In complex cases, the reward can be infinite and you can use value iteration to solve the problem. I found some paper on the topic which might help you related work "Max-Plus Matching Pursuit for Deterministic Markov Decision Processes" https://arxiv.org/pdf/1906.08524.pdf  
  > **username_0**: allright makes sense! I accepted your answer as now it is clear. I have a small followup question. How can a transition be deterministic if the environment is stochastic? So say if I move left there is 30% chance that i will transition to front instead of left. So when you say actions are irrelevant, you mean to say we dont care about the 30% chance of transitioning to front if we intend to move left?  
  > **username_1**: I think there is a misunderstanding. Transition matrix being deterministic means when you take an action you are certain that you will end up in only one state, which means the environment is deterministic. In this case (deterministic environment/transition matrix) actions are irrelevant. "So say if I move left there is 30% chance that i will transition to front instead of left." - this means the environment/transition matrix is stochastic.  
  > **username_0**: +1 This clarifies my confusion. I was confused by the underlying assumption where environment was stochastic and transition matrix was deterministic.  
  
  ---
  
  ### Answer by username_2 on 2020-06-26. Score: 0
  
  Yes , a transition model shows that our environment is stochastic in nature, and with that model we know the probability of entering a state when an action is taken.
  
  
  ---
  
  