Body: <p>I have set up a RL environment and it converges to a decent solution when using a reward function:</p>
  <p><strong><span class="math-container">$R(s_t,a_t) = fenv(s_t, a_t)$</span></strong> , where <span class="math-container">$fenv$</span> is the environment dynamics.</p>
  <p>Now, i want to change the reward function such that</p>
  <p><strong><span class="math-container">$R(s_t,a_t) = fenv(s_t, a_t)*c(s_t:s_{t-10})$</span></strong></p>
  <p>where <span class="math-container">$c(s_t:s_{t-10})$</span> is a penalty term that depends on the average performance of previous 10 timesteps. So now the reward is dependent on agent's previous states and needs information from the past. I suspect this can change the underlying problem and the MDP assumption is not valid anymore.</p>
  <p>I read about potential based reward shaping that guarantees invariance in terms of MDP but i am not sure if this transformation falls into that category.</p>
  <p>So my question is, by including information from the past states into the reward definition, do we still maintain the underlying MDP or are we solving a different problem altogether? Assuming that we don't explicitly add previous action choices in the state representation, as suggested here <a href="https://ai.stackexchange.com/a/25991/54470">https://ai.stackexchange.com/a/25991/54470</a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-10-21T16:07:31.247Z
Comments.Text: I believe this question is related and answers your concerns:
  https://ai.stackexchange.com/questions/25990/reinforcement-learning-algorithm-with-rewards-dependent-both-on-previous-action?rq=1
Comments.UserDisplayName: postnubilaphoebus
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-10-25T09:13:56.680Z
Comments.Text: Thanks @postnubilaphoebus! This confirms my suspicion about loosing the Markovian property in this case.
Comments.UserDisplayName: StarDust_08
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-12-30T15:36:40.007Z
Comments.Text: You're note asking any question explicitly. Could you please edit your post to actually ask a question? Thanks.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-01-03T14:40:16.413Z
Comments.Text: My bad @nbro! Hope the question is clear now. Thanks for your time.
Comments.UserDisplayName: StarDust_08
Tags: <reinforcement-learning><markov-decision-process><rewards><reward-shaping>
Title: MDP with a non-markovian reward function?
UserDisplayName: StarDust_08
ThreadText: # MDP with a non-markovian reward function?
  
  (Asked by: username_0 on 2022-10-21)
  
  I have set up a RL environment and it converges to a decent solution when using a reward function:
  
  **$R(s\_t,a\_t) = fenv(s\_t, a\_t)$** , where $fenv$ is the environment dynamics.
  
  Now, i want to change the reward function such that
  
  **$R(s\_t,a\_t) = fenv(s\_t, a\_t)\*c(s\_t:s\_{t-10})$**
  
  where $c(s\_t:s\_{t-10})$ is a penalty term that depends on the average performance of previous 10 timesteps. So now the reward is dependent on agent's previous states and needs information from the past. I suspect this can change the underlying problem and the MDP assumption is not valid anymore.
  
  I read about potential based reward shaping that guarantees invariance in terms of MDP but i am not sure if this transformation falls into that category.
  
  So my question is, by including information from the past states into the reward definition, do we still maintain the underlying MDP or are we solving a different problem altogether? Assuming that we don't explicitly add previous action choices in the state representation, as suggested here <https://ai.stackexchange.com/a/25991/54470>
  
  **Tags:** reinforcement-learning, markov-decision-process, rewards, reward-shaping
  
  > **Comments:**
  > **username_1**: I believe this question is related and answers your concerns: https://ai.stackexchange.com/questions/25990/reinforcement-learning-algorithm-with-rewards-dependent-both-on-previous-action?rq=1  
  > **username_0**: Thanks @username_1! This confirms my suspicion about loosing the Markovian property in this case.  
  > **username_2**: You're note asking any question explicitly. Could you please edit your post to actually ask a question? Thanks.  
  > **username_0**: My bad @username_2! Hope the question is clear now. Thanks for your time.  
  
  ---
  
  ## 0 answers
  
  