Answers.Body: <p>Your setting (of randomly dropping out reward signals) impacts <em>expected future reward</em> by multiply everything by a common factor <span class="math-container">$(1-\epsilon)$</span>.</p>
  <p>As reinforcement learning (RL) control is based on maximising expected future reward, and multiplying by a positive constant does not affect ranking of action values, all existing RL methods will cope just fine without modification. They will behave optimally in the limit of training - with all the usual caveats of course - although learning will be slower due to added variance, and value estimates lower.</p>
  <p>If the agent is allowed to observe that the reward signal is missing (as opposed to being an observed zero reward), then it could additionally estimate <span class="math-container">$\epsilon$</span>, and correct its learned value function. I would recommend this is handled at the end of training as a separate function instead of modifying <span class="math-container">$Q(s,a)$</span> during training. That is because initial estimates for <span class="math-container">$\epsilon$</span> are likely to be inaccurate and make learning even slower (at least for TD learning due to bootstrapping on less accurate values).</p>
  <p>Whether any corrected value function makes sense will depend on what it means to &quot;not observe&quot; the reward, and what the reward represents in terms of the environment and agent's goals. You may not need to know if all you care about is whether the agent is behaving optimally.</p>
  <p>The setting becomes more complex if <span class="math-container">$\epsilon$</span> is made a function of <span class="math-container">$a$</span>, <span class="math-container">$s$</span> or both, and solutions will be different depending on whether this function is known or unknown.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-30T16:43:25.790Z
Answers.Text: Thank you very much! You are totally right.. Yeah, the way I have formulated the problem is actually very trivial, as you correctly noticed.  As you pointed out, the situation is less trivial if $\varepsilon$ depends on $s$ and/or $a$. 
  Do you think there could be any other meaningful notion of "partial observability" for the reward function?
Answers.UserDisplayName: Onil90
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-30T10:10:11.253Z
Answers.LastActivityDate: 2021-11-30T10:16:24.250Z
Answers.LastEditDate: 2021-11-30T10:16:24.250Z
Answers.UserDisplayName: Neil Slater
Body: <p>Typically, a Reinforcement Learning learning problem is formalized as finding an optimal policy for a Markov Decision Process (MDP). In many real-life situations, however, an agent can only get partial information from the environment. For example, <a href="https://en.wikipedia.org/wiki/Partially_observable_Markov_decision_process" rel="nofollow noreferrer">Partially Observable MDPs</a> are used to model the case where the agent does not fully observe the state.</p>
  <p>I was wondering whether there is any well-established formalism for the case where the agent does not fully observe the reward signal.</p>
  <p>In particular, I am thinking about the case where for every state-action pair <span class="math-container">$(s, a)$</span> the agent receives the reward <span class="math-container">$R(s, a)$</span> with probability <span class="math-container">$1 - \varepsilon$</span> and does not receive anything with probability <span class="math-container">$\varepsilon$</span>. Of course, in principle, this setting can be thought as a regular MDP with a stochastic reward, but here I would like the agent to behave optimally w.r.t. to <span class="math-container">$R$</span>.</p>
  <p>I would really appreciate if you could point some relevant literature to me!</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-12-01T10:13:00.077Z
Comments.Text: This does not directly answer your question (I think), but there are ways of dealing with problems where the reward function (RF) is not specified or it's not easy to specify it correctly. The basic idea is to learn the RF itself (this is sometimes known as _inverse RL_ or _reward learning_, especially when you are given an optimal policy: that's why it's called IRL because, usually, in an RL problem, you want to find the policy from the RF), but you can also use _behaviour cloning_ (i.e. you don't use a reward function at all, and learn a policy from examples taken from another policy).
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-12-01T10:29:43.493Z
Comments.Text: @nbro Thanks for the comment! Good point! In fact I asked this question precisely because I am reading about IRL at the moment :)
  Oversimplifying, RL deals with MDP with observable reward function and IRL deals with the case where the reward function is missing.  I was wondering whether some intermediate scenarios have been studied..
Comments.UserDisplayName: Onil90
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-12-01T11:00:51.767Z
Comments.Text: I am not sure if it's intermediate, but in the paper "[Inverse Reward Design](http://people.eecs.berkeley.edu/~russell/papers/nips17-ird.pdf)", if I remember correctly, they formulate the problem of learning a "correct" function from "observed RFs" (i.e. non-optimal or partial/incomplete ones) in a Bayesian/probabilistic setting. So, in a way, if I remember correctly, they kind of combine IRL with RL because they need to solve the RL problem too (but I could be recalling that wrongly, so you may want to check the paper).
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-12-01T11:02:38.483Z
Comments.Text: Let me know if this answers your question so that I eventually give a formal answer below with this info.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-12-01T15:01:42.457Z
Comments.Text: @nbro Thanks a lot for the paper! I'll have a look and I'll let you know!
Comments.UserDisplayName: Onil90
Tags: <reinforcement-learning><reference-request><markov-decision-process><reward-functions><pomdp>
Title: Is there a mathematical formalism to deal with a missing reward signal?
UserDisplayName: Onil90
ThreadText: # Is there a mathematical formalism to deal with a missing reward signal?
  
  (Asked by: username_0 on 2021-11-30)
  
  Typically, a Reinforcement Learning learning problem is formalized as finding an optimal policy for a Markov Decision Process (MDP). In many real-life situations, however, an agent can only get partial information from the environment. For example, [Partially Observable MDPs](https://en.wikipedia.org/wiki/Partially_observable_Markov_decision_process) are used to model the case where the agent does not fully observe the state.
  
  I was wondering whether there is any well-established formalism for the case where the agent does not fully observe the reward signal.
  
  In particular, I am thinking about the case where for every state-action pair $(s, a)$ the agent receives the reward $R(s, a)$ with probability $1 - \varepsilon$ and does not receive anything with probability $\varepsilon$. Of course, in principle, this setting can be thought as a regular MDP with a stochastic reward, but here I would like the agent to behave optimally w.r.t. to $R$.
  
  I would really appreciate if you could point some relevant literature to me!
  
  **Tags:** reinforcement-learning, reference-request, markov-decision-process, reward-functions, pomdp
  
  > **Comments:**
  > **username_1**: This does not directly answer your question (I think), but there are ways of dealing with problems where the reward function (RF) is not specified or it's not easy to specify it correctly. The basic idea is to learn the RF itself (this is sometimes known as _inverse RL_ or _reward learning_, especially when you are given an optimal policy: that's why it's called IRL because, usually, in an RL problem, you want to find the policy from the RF), but you can also use _behaviour cloning_ (i.e. you don't use a reward function at all, and learn a policy from examples taken from another policy).  
  > **username_0**: @username_1 Thanks for the comment! Good point! In fact I asked this question precisely because I am reading about IRL at the moment :) Oversimplifying, RL deals with MDP with observable reward function and IRL deals with the case where the reward function is missing.  I was wondering whether some intermediate scenarios have been studied..  
  > **username_1**: I am not sure if it's intermediate, but in the paper "[Inverse Reward Design](http://people.eecs.berkeley.edu/~russell/papers/nips17-ird.pdf)", if I remember correctly, they formulate the problem of learning a "correct" function from "observed RFs" (i.e. non-optimal or partial/incomplete ones) in a Bayesian/probabilistic setting. So, in a way, if I remember correctly, they kind of combine IRL with RL because they need to solve the RL problem too (but I could be recalling that wrongly, so you may want to check the paper).  
  > **username_1**: Let me know if this answers your question so that I eventually give a formal answer below with this info.  
  > **username_0**: @username_1 Thanks a lot for the paper! I'll have a look and I'll let you know!  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-11-30. Score: 2
  
  Your setting (of randomly dropping out reward signals) impacts *expected future reward* by multiply everything by a common factor $(1-\epsilon)$.
  
  As reinforcement learning (RL) control is based on maximising expected future reward, and multiplying by a positive constant does not affect ranking of action values, all existing RL methods will cope just fine without modification. They will behave optimally in the limit of training - with all the usual caveats of course - although learning will be slower due to added variance, and value estimates lower.
  
  If the agent is allowed to observe that the reward signal is missing (as opposed to being an observed zero reward), then it could additionally estimate $\epsilon$, and correct its learned value function. I would recommend this is handled at the end of training as a separate function instead of modifying $Q(s,a)$ during training. That is because initial estimates for $\epsilon$ are likely to be inaccurate and make learning even slower (at least for TD learning due to bootstrapping on less accurate values).
  
  Whether any corrected value function makes sense will depend on what it means to "not observe" the reward, and what the reward represents in terms of the environment and agent's goals. You may not need to know if all you care about is whether the agent is behaving optimally.
  
  The setting becomes more complex if $\epsilon$ is made a function of $a$, $s$ or both, and solutions will be different depending on whether this function is known or unknown.
  
  > **Comments:**
  > **username_0**: Thank you very much! You are totally right.. Yeah, the way I have formulated the problem is actually very trivial, as you correctly noticed.  As you pointed out, the situation is less trivial if $\varepsilon$ depends on $s$ and/or $a$.  Do you think there could be any other meaningful notion of "partial observability" for the reward function?  
  
  ---
  
  