Answers.Body: <blockquote>
    <p>My question is, would <span class="math-container">$r_1 =r_2$</span>?</p>
  </blockquote>
  
  <p>That's usually up to you as the designer of the system.</p>
  
  <p>Usually when you declare that you have "a deterministic environment", you imply that both <span class="math-container">$s'$</span> and <span class="math-container">$r$</span> are fixed values depending on <span class="math-container">$(s,a)$</span>. So in your examples, you would expect your observations to also have <span class="math-container">$r_1 = r_2$</span></p>
  
  <p>However, it is <em>possible</em> to define a MDP where transition to state <span class="math-container">$s'$</span> is deterministic, but <span class="math-container">$r$</span> is not. For instance, you could define reward in a game equal to the sum of a number of dice rolled, with better rewards (on average) resulting in more dice. This is still a valid MDP and can be solved using RL techniques.</p>
  
  <p>A real-world example of this might be managing a queue of work, where you want to minimise lead time, but don't know for certain how long each task will take. Your state progression moves deterministically - you have a queue of pending tasks, current tasks and workers, and assigning a task to a worker is completely deterministic. However, you don't know how efficiently tasks will be performed until after they are done, so you don't know the reward perfectly from the assignment (whether you can treat this as random or hidden state is a more complex issue - it is often pragmatic to treat such unknown data as random though).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-12T10:46:35.830Z
Answers.LastActivityDate: 2019-02-13T07:43:06.287Z
Answers.LastEditDate: 2019-02-13T07:43:06.287Z
Answers.UserDisplayName: Neil Slater
Body: <p>Suppose we have a deterministic environment where knowing <span class="math-container">$s,a$</span> determines <span class="math-container">$s'$</span>. Is it possible to get two different rewards <span class="math-container">$r\neq r'$</span> in some state <span class="math-container">$s_{\text{fixed}}$</span>? Assume that <span class="math-container">$s_{\text{fixed}}$</span> is a fixed state I get to after taking the action <span class="math-container">$a$</span>. Note that we can have situations where in multiple iterations we have: <span class="math-container">$$(s,a) \to (s_1, r_1) \\ (s,a) \to (s_{\text{fixed}}, r_1) \\ (s,a) \to (s_{\text{fixed}}, r_2) \\ (s,a) \to (s_3, r_3) \\ \vdots$$</span></p>
  
  <p>My question is, would <span class="math-container">$r_1 =r_2$</span>?</p>
  
Tags: <reinforcement-learning><markov-decision-process><reward-design><reward-functions>
Title: Can the rewards be stochastic when the transition model is deterministic?
UserDisplayName: rewardbabe
ThreadText: # Can the rewards be stochastic when the transition model is deterministic?
  
  (Asked by: username_0 on 2019-02-12)
  
  Suppose we have a deterministic environment where knowing $s,a$ determines $s'$. Is it possible to get two different rewards $r\neq r'$ in some state $s\_{\text{fixed}}$? Assume that $s\_{\text{fixed}}$ is a fixed state I get to after taking the action $a$. Note that we can have situations where in multiple iterations we have: $$(s,a) \to (s\_1, r\_1) \\ (s,a) \to (s\_{\text{fixed}}, r\_1) \\ (s,a) \to (s\_{\text{fixed}}, r\_2) \\ (s,a) \to (s\_3, r\_3) \\ \vdots$$
  
  My question is, would $r\_1 =r\_2$?
  
  **Tags:** reinforcement-learning, markov-decision-process, reward-design, reward-functions
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-02-12. Score: 4
  
  > My question is, would $r\_1 =r\_2$?
  
  That's usually up to you as the designer of the system.
  
  Usually when you declare that you have "a deterministic environment", you imply that both $s'$ and $r$ are fixed values depending on $(s,a)$. So in your examples, you would expect your observations to also have $r\_1 = r\_2$
  
  However, it is *possible* to define a MDP where transition to state $s'$ is deterministic, but $r$ is not. For instance, you could define reward in a game equal to the sum of a number of dice rolled, with better rewards (on average) resulting in more dice. This is still a valid MDP and can be solved using RL techniques.
  
  A real-world example of this might be managing a queue of work, where you want to minimise lead time, but don't know for certain how long each task will take. Your state progression moves deterministically - you have a queue of pending tasks, current tasks and workers, and assigning a task to a worker is completely deterministic. However, you don't know how efficiently tasks will be performed until after they are done, so you don't know the reward perfectly from the assignment (whether you can treat this as random or hidden state is a more complex issue - it is often pragmatic to treat such unknown data as random though).
  
  
  ---
  
  