Body: <p>In a discrete time system, an environment has an unknown reward probability <span class="math-container">$p(r|s,a)$</span>. However, the transition probability <span class="math-container">$p(s'\mid s,a)$</span> is deterministic.</p>
  <p>In my case, the reward for the same <strong>action</strong> can be drawn as a staircase line: it goes up gradually most of the time, but can occasionally go down when a special event happens. The <strong>state</strong> can capture the time elapsed since the last time the observed special event happened to each action object (some events couldn't be observed).</p>
  <p>I know the reward function in RL could be represented as a probability. But <strong>I don't know if it's stationary or not</strong>.</p>
  <p>Is this normal in reinforcement learning problems? Can I solve it using TD learning or actor-critic algorithms? Or POMDP could be the solution? Or maybe regret minimization for incomplete information could work too. In which direction should I go?</p>
  <p>I'm new to RL. If you want more details, you can see my other <a href="https://ai.stackexchange.com/questions/35809/reinforcement-learning-with-changing-rewards">question</a>.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-07T11:39:11.990Z
Comments.Text: One thing to bear in mind is that a non-stationary environment may not be fully optimised in terms of always being able to follow an optimal policy. You probably need to set your goals on some other target, such as minimising regret, or spending some percentage of time behaving optimally. So you may want to clarify what you mean by "solved"
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-07T20:12:04.350Z
Comments.Text: Note that $p(s',r \mid s,a)$ is not what you call "a stochastic reward probability". If you consider that a probability, then that's the probability of getting $s'$ and $r'$ at the same time. Typically, that notation refers to the model of the environment, which includes both the transition dynamics and reward function (which can or not be stochastic). Maybe you mean to use $r(s, a)$ or $p(r \mid s, a)$?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-07T20:13:30.870Z
Comments.Text: Also, can you explain this conclusion "_Because the  $p(s',r \mid s,a)$  is stochastic, the reward could be stationary or non-stationary._"? That does not have to be the case. The rewards can be stochastic and still be stationary, i.e. the $p(r \mid s, a)$ does not change over time. Please, **edit your post to fix these issues**.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-09T00:23:08.733Z
Comments.Text: @nbro Can see my edited post above. The linked question is edited too.  You were right about transition probability and reward function. I differentiate them in the post.
Comments.UserDisplayName: Zhenzhen Gong
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-09T11:26:07.940Z
Comments.Text: If you don't know the reward function, you cannot apply RL algorithms. You could use imitation learning, though. Can you clarify whether you really know or not the reward, i.e. even if it's just a probability distribution? You seem to know how the reward oscillates but you say that you don't know $p(r \mid s, a)$. Those statements seem contradictory, but you also don't explain in enough detail how the rewards behave. BTW I added the to the title the word "stochastic". That is only correct if your rewards are really stochastic, i.e. they are not always the same for the same state and action.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-10T10:09:46.817Z
Comments.Text: @nbro I'm sorry but I don't understand why it's contradictory. (1) I know the rewards oscillate because I did the simulation and saw the results. But I don't know the reward function because I don't know the statistical regularity.  (2) What do you mean by "If you don't know the reward function, you cannot apply RL algorithms"? As I learned, the reward is obtained from the interaction with the environment.  (3) I did a demo test and force the action with some regularity. It turns out the result is better than random. Maybe this can be used in imitation learning as an expert?
Comments.UserDisplayName: Zhenzhen Gong
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-10T11:57:04.197Z
Comments.Text: "As I learned, the reward is obtained from the interaction with the environment." - You get the rewards while interacting with the environment, but there must be some kind of function $r(s, a)$ that given a state and action returns you reward, otherwise, you cannot apply RL. Like in supervised learning, you need to have the labels to train the model. Note that there's a difference between "reward" and "return" - the return is not known in advance - maybe that's what is confusing you. I think our website has one or more questions about that, so can check them out.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-10T11:58:30.877Z
Comments.Text: Imitation learning can be used when you don't have rewards but have trajectories of optimal/good behaviour. This post has already one too far. If you have question about a specific topic, e.g. "What is imitation learning in RL", ask that as a separate question.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-13T04:34:15.317Z
Comments.Text: @nbro Thank you. 'but there must be some kind of function (,) that given a state and action returns you reward' - Of course, I have the function to get the reward. But I don't have the probability to know $p(r|s, a)$. That's why I can't know if it's stationary or not. So, why do you say 'you cannot apply RL'? Like in Supervised learning, I must have labels doesn't mean I know the function of the label.
Comments.UserDisplayName: Zhenzhen Gong
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-13T11:00:19.087Z
Comments.Text: If you define $r(s, a)$ as the expected reward, then indeed you may not know $p(r \mid s, a)$, but you know it's stationary because otherwise $r(s, a)$ would give you a different result every time (unless it's non-stationary but looks stationary from the perspective of $r(s, a)$, but that's unlikely, I suppose).
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><rewards><algorithm-request><reward-functions><reward-design>
Title: How to solve a reinforcement learning problem with a stochastic reward function?
UserDisplayName: Zhenzhen Gong
ThreadText: # How to solve a reinforcement learning problem with a stochastic reward function?
  
  (Asked by: username_0 on 2022-06-07)
  
  In a discrete time system, an environment has an unknown reward probability $p(r|s,a)$. However, the transition probability $p(s'\mid s,a)$ is deterministic.
  
  In my case, the reward for the same **action** can be drawn as a staircase line: it goes up gradually most of the time, but can occasionally go down when a special event happens. The **state** can capture the time elapsed since the last time the observed special event happened to each action object (some events couldn't be observed).
  
  I know the reward function in RL could be represented as a probability. But **I don't know if it's stationary or not**.
  
  Is this normal in reinforcement learning problems? Can I solve it using TD learning or actor-critic algorithms? Or POMDP could be the solution? Or maybe regret minimization for incomplete information could work too. In which direction should I go?
  
  I'm new to RL. If you want more details, you can see my other [question](https://ai.stackexchange.com/questions/35809/reinforcement-learning-with-changing-rewards).
  
  **Tags:** reinforcement-learning, rewards, algorithm-request, reward-functions, reward-design
  
  > **Comments:**
  > **username_1**: One thing to bear in mind is that a non-stationary environment may not be fully optimised in terms of always being able to follow an optimal policy. You probably need to set your goals on some other target, such as minimising regret, or spending some percentage of time behaving optimally. So you may want to clarify what you mean by "solved"  
  > **username_2**: Note that $p(s',r \mid s,a)$ is not what you call "a stochastic reward probability". If you consider that a probability, then that's the probability of getting $s'$ and $r'$ at the same time. Typically, that notation refers to the model of the environment, which includes both the transition dynamics and reward function (which can or not be stochastic). Maybe you mean to use $r(s, a)$ or $p(r \mid s, a)$?  
  > **username_2**: Also, can you explain this conclusion "_Because the  $p(s',r \mid s,a)$  is stochastic, the reward could be stationary or non-stationary._"? That does not have to be the case. The rewards can be stochastic and still be stationary, i.e. the $p(r \mid s, a)$ does not change over time. Please, **edit your post to fix these issues**.  
  > **username_0**: @username_2 Can see my edited post above. The linked question is edited too.  You were right about transition probability and reward function. I differentiate them in the post.  
  > **username_2**: If you don't know the reward function, you cannot apply RL algorithms. You could use imitation learning, though. Can you clarify whether you really know or not the reward, i.e. even if it's just a probability distribution? You seem to know how the reward oscillates but you say that you don't know $p(r \mid s, a)$. Those statements seem contradictory, but you also don't explain in enough detail how the rewards behave. BTW I added the to the title the word "stochastic". That is only correct if your rewards are really stochastic, i.e. they are not always the same for the same state and action.  
  > **username_0**: @username_2 I'm sorry but I don't understand why it's contradictory. (1) I know the rewards oscillate because I did the simulation and saw the results. But I don't know the reward function because I don't know the statistical regularity.  (2) What do you mean by "If you don't know the reward function, you cannot apply RL algorithms"? As I learned, the reward is obtained from the interaction with the environment.  (3) I did a demo test and force the action with some regularity. It turns out the result is better than random. Maybe this can be used in imitation learning as an expert?  
  > **username_2**: "As I learned, the reward is obtained from the interaction with the environment." - You get the rewards while interacting with the environment, but there must be some kind of function $r(s, a)$ that given a state and action returns you reward, otherwise, you cannot apply RL. Like in supervised learning, you need to have the labels to train the model. Note that there's a difference between "reward" and "return" - the return is not known in advance - maybe that's what is confusing you. I think our website has one or more questions about that, so can check them out.  
  > **username_2**: Imitation learning can be used when you don't have rewards but have trajectories of optimal/good behaviour. This post has already one too far. If you have question about a specific topic, e.g. "What is imitation learning in RL", ask that as a separate question.  
  > **username_0**: @username_2 Thank you. 'but there must be some kind of function (,) that given a state and action returns you reward' - Of course, I have the function to get the reward. But I don't have the probability to know $p(r|s, a)$. That's why I can't know if it's stationary or not. So, why do you say 'you cannot apply RL'? Like in Supervised learning, I must have labels doesn't mean I know the function of the label.  
  > **username_2**: If you define $r(s, a)$ as the expected reward, then indeed you may not know $p(r \mid s, a)$, but you know it's stationary because otherwise $r(s, a)$ would give you a different result every time (unless it's non-stationary but looks stationary from the perspective of $r(s, a)$, but that's unlikely, I suppose).  
  
  ---
  
  ## 0 answers
  
  