Answers.Body: <p>As explained <a href="https://ai.stackexchange.com/q/10442/2444">here</a>, I can write <span class="math-container">$R(s,a) = R(s)\  \forall a$</span> since the reward of my specific MDP is dependent exclusively to the state <span class="math-container">$s$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-31T19:04:27.750Z
Answers.Text: Just a clarification, if you're trying to reproduce a certain paper that uses $R(s, a)$, why do you say that in your MDP the reward function only depends on $s$? I suppose you're testing their approach in a simpler environment. Right? Is the code you're using available?
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-01T10:13:01.783Z
Answers.Text: Yes exactly, since the paper doesn't present an example environment, I'm using the one from [here](https://www.aaai.org/Papers/IJCAI/2007/IJCAI07-416.pdf), the code isn't available yet (I couldn't find any implementation online so I'm writing my own)
Answers.UserDisplayName: ИванКарамазов
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-31T16:47:58.117Z
Answers.LastActivityDate: 2021-01-31T19:03:00.123Z
Answers.LastEditDate: 2021-01-31T19:03:00.123Z
Answers.UserDisplayName: ИванКарамазов
Body: <p>I'm trying to implement a research paper, as explained in <a href="https://ai.stackexchange.com/questions/25993/what-is-the-dimensionality-of-these-derivatives-in-the-paper-active-learning-fo">this</a> other post, here the author of the paper assumed R as a function of both states and actions, while the code (and the MDP) I'm using to test this algorithm assumes R as a function of only states.</p>
  <p>My question is:</p>
  <p>Given <span class="math-container">$\mathcal{X}$</span> as the set of states of an MDP and <span class="math-container">$\mathcal{A}$</span> as the set of actions of an MDP. Supposing I have four states (<span class="math-container">$1$</span>,<span class="math-container">$2$</span>,<span class="math-container">$3$</span>,<span class="math-container">$4$</span>), two actions <span class="math-container">$a$</span> and <span class="math-container">$b$</span> and a reward function <span class="math-container">$R: \mathcal{X}\to\mathbb{R}$</span> s.t.</p>
  <p><span class="math-container">$R(1) = 0$</span></p>
  <p><span class="math-container">$R(2) = 0$</span></p>
  <p><span class="math-container">$R(3) = 0$</span></p>
  <p><span class="math-container">$R(4) = 1$</span></p>
  <p>If I need to change the current reward function to a new reward function <span class="math-container">$R:\mathcal{X}\ \times \mathcal{A} \to\mathbb{R}$</span> is it ok to compute it as <span class="math-container">$\forall a,R(s,a) = R(s)$</span>?</p>
  <p><span class="math-container">$R(1,a) = 0$</span></p>
  <p><span class="math-container">$R(1,b) = 0$</span></p>
  <p><span class="math-container">$R(2,a) = 0$</span></p>
  <p><span class="math-container">$R(2,b) = 0$</span></p>
  <p><span class="math-container">$R(3,a) = 0$</span></p>
  <p><span class="math-container">$R(3,b) = 0$</span></p>
  <p><span class="math-container">$R(4,a) = 1$</span></p>
  <p><span class="math-container">$R(4,b) = 1$</span></p>
  <p>More generally, what's the correct way of generalising a reward function
  <span class="math-container">$R: \mathcal{X}\to\mathbb{R}$</span> to a reward function <span class="math-container">$R:\mathcal{X}\ \times \mathcal{A} \to\mathbb{R}$</span>?</p>
  
Tags: <reinforcement-learning><markov-decision-process><reward-functions>
Title: How can I go from $R(s)$ to $R(s,a)$ in this specific MDP?
UserDisplayName: ИванКарамазов
ThreadText: # How can I go from $R(s)$ to $R(s,a)$ in this specific MDP?
  
  (Asked by: username_0 on 2021-01-27)
  
  I'm trying to implement a research paper, as explained in [this](https://ai.stackexchange.com/questions/25993/what-is-the-dimensionality-of-these-derivatives-in-the-paper-active-learning-fo) other post, here the author of the paper assumed R as a function of both states and actions, while the code (and the MDP) I'm using to test this algorithm assumes R as a function of only states.
  
  My question is:
  
  Given $\mathcal{X}$ as the set of states of an MDP and $\mathcal{A}$ as the set of actions of an MDP. Supposing I have four states ($1$,$2$,$3$,$4$), two actions $a$ and $b$ and a reward function $R: \mathcal{X}\to\mathbb{R}$ s.t.
  
  $R(1) = 0$
  
  $R(2) = 0$
  
  $R(3) = 0$
  
  $R(4) = 1$
  
  If I need to change the current reward function to a new reward function $R:\mathcal{X}\ \times \mathcal{A} \to\mathbb{R}$ is it ok to compute it as $\forall a,R(s,a) = R(s)$?
  
  $R(1,a) = 0$
  
  $R(1,b) = 0$
  
  $R(2,a) = 0$
  
  $R(2,b) = 0$
  
  $R(3,a) = 0$
  
  $R(3,b) = 0$
  
  $R(4,a) = 1$
  
  $R(4,b) = 1$
  
  More generally, what's the correct way of generalising a reward function
  $R: \mathcal{X}\to\mathbb{R}$ to a reward function $R:\mathcal{X}\ \times \mathcal{A} \to\mathbb{R}$?
  
  **Tags:** reinforcement-learning, markov-decision-process, reward-functions
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2021-01-31. Score: 0 | Accepted answer
  
  As explained [here](https://ai.stackexchange.com/q/10442/2444), I can write $R(s,a) = R(s)\ \forall a$ since the reward of my specific MDP is dependent exclusively to the state $s$.
  
  > **Comments:**
  > **username_1**: Just a clarification, if you're trying to reproduce a certain paper that uses $R(s, a)$, why do you say that in your MDP the reward function only depends on $s$? I suppose you're testing their approach in a simpler environment. Right? Is the code you're using available?  
  > **username_0**: Yes exactly, since the paper doesn't present an example environment, I'm using the one from [here](https://www.aaai.org/Papers/IJCAI/2007/IJCAI07-416.pdf), the code isn't available yet (I couldn't find any implementation online so I'm writing my own)  
  
  ---
  
  