Answers.Body: <p>Maybe not clear from the other answer, but no, the reward depends usually also on the state you end up, which is not deterministic by definition of MDPs, and even in that case, the reward can be noisy.</p>
  <p>Take the simplest case of Bandits, where you have a single state, and you can pull one of N levers, then when you pull a lever, you observe a reward that might change (like a lottery)</p>
  <p>So, long story short, you can define any reward as a distribution, aka, no they are not deterministic</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-12T21:29:03.460Z
Answers.LastActivityDate: 2023-09-12T21:29:03.460Z
Answers.UserDisplayName: Alberto
Answers.Body: <p>The RL framework assumes only that the reward function depends on the current state, the selected action, and the next state:</p>
  <p><span class="math-container">$$\mathcal{R}(s_t, a_t, s_{t+1})$$</span></p>
  <p>It can be deterministic but it can also be probabilistic. As long as it depends only on <span class="math-container">$(s_t, a_t, s_{t+1})$</span> its fine.</p>
  <p>You can see equations (3.2) and (3.5) from the <a href="http://incompleteideas.net/book/RLbook2020.pdf#page70" rel="nofollow noreferrer">Sutton and Barto book</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-12T20:39:51.780Z
Answers.Text: May be worth saying, for clarity for the probabilistic result, that $\mathcal{R}(s_t, a_t, s_{t+1})$ returns the same *distribution* for the same arguments, where the deterministic version mentioned by OP would only return the same single *value*. Technically it doesn't even need to be the same distribution, just the same mean/expected immediate reward, but that would be an unusual MDP design.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-12T19:15:44.440Z
Answers.LastActivityDate: 2023-09-12T19:15:44.440Z
Answers.UserDisplayName: pi-tau
Body: <p>I am designing my own environment for a specific problem and I am thinking of the reward function for it. In some RL algos it is common to learn the reward that is associated with taking an action given the state. I therefore expect that a reward should always be the same when you encounter the exact same state and take the exact same action. Is this a constraint that really exists or is this a non-issue and the RL algorithm can deal with this variance in rewards received? I would like to center my rewards around 0.0 and it would be more intuitive to shape the rewards like this, but I am worried that because of the variance in rewards it might confuse the model.</p>
  
Tags: <reinforcement-learning><rewards><reward-shaping>
Title: One to one relation between state + action -> reward
UserDisplayName: Erik Storm
ThreadText: # One to one relation between state + action -> reward
  
  (Asked by: username_0 on 2023-09-12)
  
  I am designing my own environment for a specific problem and I am thinking of the reward function for it. In some RL algos it is common to learn the reward that is associated with taking an action given the state. I therefore expect that a reward should always be the same when you encounter the exact same state and take the exact same action. Is this a constraint that really exists or is this a non-issue and the RL algorithm can deal with this variance in rewards received? I would like to center my rewards around 0.0 and it would be more intuitive to shape the rewards like this, but I am worried that because of the variance in rewards it might confuse the model.
  
  **Tags:** reinforcement-learning, rewards, reward-shaping
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2023-09-12. Score: 3
  
  Maybe not clear from the other answer, but no, the reward depends usually also on the state you end up, which is not deterministic by definition of MDPs, and even in that case, the reward can be noisy.
  
  Take the simplest case of Bandits, where you have a single state, and you can pull one of N levers, then when you pull a lever, you observe a reward that might change (like a lottery)
  
  So, long story short, you can define any reward as a distribution, aka, no they are not deterministic
  
  
  ---
  
  ### Answer by username_2 on 2023-09-12. Score: 3
  
  The RL framework assumes only that the reward function depends on the current state, the selected action, and the next state:
  
  $$\mathcal{R}(s\_t, a\_t, s\_{t+1})$$
  
  It can be deterministic but it can also be probabilistic. As long as it depends only on $(s\_t, a\_t, s\_{t+1})$ its fine.
  
  You can see equations (3.2) and (3.5) from the [Sutton and Barto book](http://incompleteideas.net/book/RLbook2020.pdf#page70).
  
  > **Comments:**
  > **username_3**: May be worth saying, for clarity for the probabilistic result, that $\mathcal{R}(s_t, a_t, s_{t+1})$ returns the same *distribution* for the same arguments, where the deterministic version mentioned by OP would only return the same single *value*. Technically it doesn't even need to be the same distribution, just the same mean/expected immediate reward, but that would be an unusual MDP design.  
  
  ---
  
  