Body: <p>In the context of RL, say I'm performing Value Iteration on a reward function R1. And the converged optimal policy is P1 and values are V1.</p>
  <p>Then, let's say I set rewards to be R2=V1 and perform value iteration from scratch on these rewards and get a value V2 and policy P2.</p>
  <p>Is P2 == P1 ?</p>
  <p>Rewards can be just dependent on just the states as well, for example, R(s), it doesn't necessarily have to depend on the action being taken in that state.</p>
  <p>For MDP = <span class="math-container">$&lt;S, T, A, R&gt;$</span>, S are set of states, T is transition function as <span class="math-container">$S \times A \times S \rightarrow [0,1]$</span>, A is set of actions, R is the reward function <span class="math-container">$S \rightarrow \mathbb{R}$</span>.</p>
  <p>Formally, <span class="math-container">$MDP1 = &lt;S, T(s,a,s'), A, R(s) = R1(s)&gt;$</span> yields optimal value <span class="math-container">$v*(s) = v_1$</span> and policy <span class="math-container">$\pi_1$</span>,
  and <span class="math-container">$MDP2 = &lt;S, T(s,a,s'), A, R(s) = v_1(s)&gt;$</span> yields optimal policy <span class="math-container">$\pi_2$</span>.</p>
  <p>Would <span class="math-container">$\pi_2(s) = \pi_1(s)$</span>? when <span class="math-container">$\gamma$</span> = 0, 0 &lt; <span class="math-container">$\gamma$</span> &lt; 1, <span class="math-container">$\gamma$</span> = 1.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-11-09T07:23:19.413Z
Comments.Text: The reward function is usually dependent on the state $s$ and action $a$, and possibly the next state $s'$, that is either $R(s,a)$ or $R(s',a,s)$. The value function of any policy $\pi$ is a function of the state, that is $V^{\pi}(s)$. How do you set the second reward function to equal first value function in your case?
Comments.UserDisplayName: mikkola
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-11-09T08:23:53.670Z
Comments.Text: You could say that reward function $R_1$ is dependent only on either $s$ or $s'$, and that $R_2 = V^*(s)$ (or same function of $s'$, matching how you have defined $R_1$). This would answer mikkola's question, but it does show that any result cannot be general to all MDPs.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-11-09T13:43:31.207Z
Comments.Text: Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.
Comments.UserDisplayName: Community
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-11-09T17:44:51.943Z
Comments.Text: Hi @mikkola I have updated my question to be more formal. Just to reiterate on your point, R(s) is perfectly valid as you can marginalize the effect of actions from R(s,a) or marginalize a, s' in R(s,a,s'). Therefore, R and V are both defined on $S \rightarrow \mathbb{R}$.
Comments.UserDisplayName: famishedrover
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-11-09T17:53:11.660Z
Comments.Text: @mikkkola I may have misunderstood the conversion from R(s,a) to R(s), but regardless the question I want to ask is with respect to R(s).
Comments.UserDisplayName: famishedrover
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-11-14T10:34:21.480Z
Comments.Text: Setting $R_2(s)=v_1^*(s)$ amounts to saying per-state reward in the second MDP will equal the expected long-term optimal reward starting from said state in the first MDP. My gut feeling says probably $\pi_2^* \equiv \pi_1^*$, but I have not yet been able to prove this, nor to come up with a counterexample.
Comments.UserDisplayName: mikkola
Tags: <reinforcement-learning><markov-decision-process><reward-functions><value-iteration><policy-iteration>
Title: Would the optimal policy remain same, if I replace R with V*?
UserDisplayName: famishedrover
ThreadText: # Would the optimal policy remain same, if I replace R with V*?
  
  (Asked by: username_0 on 2022-11-09)
  
  In the context of RL, say I'm performing Value Iteration on a reward function R1. And the converged optimal policy is P1 and values are V1.
  
  Then, let's say I set rewards to be R2=V1 and perform value iteration from scratch on these rewards and get a value V2 and policy P2.
  
  Is P2 == P1 ?
  
  Rewards can be just dependent on just the states as well, for example, R(s), it doesn't necessarily have to depend on the action being taken in that state.
  
  For MDP = $<S, T, A, R>$, S are set of states, T is transition function as $S \times A \times S \rightarrow [0,1]$, A is set of actions, R is the reward function $S \rightarrow \mathbb{R}$.
  
  Formally, $MDP1 = <S, T(s,a,s'), A, R(s) = R1(s)>$ yields optimal value $v\*(s) = v\_1$ and policy $\pi\_1$,
  and $MDP2 = <S, T(s,a,s'), A, R(s) = v\_1(s)>$ yields optimal policy $\pi\_2$.
  
  Would $\pi\_2(s) = \pi\_1(s)$? when $\gamma$ = 0, 0 < $\gamma$ < 1, $\gamma$ = 1.
  
  **Tags:** reinforcement-learning, markov-decision-process, reward-functions, value-iteration, policy-iteration
  
  > **Comments:**
  > **username_1**: The reward function is usually dependent on the state $s$ and action $a$, and possibly the next state $s'$, that is either $R(s,a)$ or $R(s',a,s)$. The value function of any policy $\pi$ is a function of the state, that is $V^{\pi}(s)$. How do you set the second reward function to equal first value function in your case?  
  > **username_2**: You could say that reward function $R_1$ is dependent only on either $s$ or $s'$, and that $R_2 = V^*(s)$ (or same function of $s'$, matching how you have defined $R_1$). This would answer username_1's question, but it does show that any result cannot be general to all MDPs.  
  > **username_3**: Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.  
  > **username_0**: Hi @username_1 I have updated my question to be more formal. Just to reiterate on your point, R(s) is perfectly valid as you can marginalize the effect of actions from R(s,a) or marginalize a, s' in R(s,a,s'). Therefore, R and V are both defined on $S \rightarrow \mathbb{R}$.  
  > **username_0**: @mikkkola I may have misunderstood the conversion from R(s,a) to R(s), but regardless the question I want to ask is with respect to R(s).  
  > **username_1**: Setting $R_2(s)=v_1^*(s)$ amounts to saying per-state reward in the second MDP will equal the expected long-term optimal reward starting from said state in the first MDP. My gut feeling says probably $\pi_2^* \equiv \pi_1^*$, but I have not yet been able to prove this, nor to come up with a counterexample.  
  
  ---
  
  ## 0 answers
  
  