Answers.Body: <p>After doing some further reading, it turns out that negative rewards <em>are</em> an assumption for this distribution to hold. However, the author notes that as long as you don't receive a reward of infinity for any action then it is possible to re-scale your rewards by subtracting the maximum value of your potential rewards so that they are always negative.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-23T18:30:05.000Z
Answers.LastActivityDate: 2020-12-23T18:30:05.000Z
Answers.UserDisplayName: David
Body: <p>In section 2 the paper <a href="https://arxiv.org/pdf/1805.00909.pdf" rel="nofollow noreferrer">Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review</a> the author is discussing formulating the RL problem as a probabilistic graphical model. They introduce a binary optimality variable <span class="math-container">$\mathcal{O}_t$</span> which denotes whether time step <span class="math-container">$t$</span> was optimal (1 if so, 0 otherwise). They then define the probability that this random variable equals 1 to be</p>
  <p><span class="math-container">$$\mathbb{P}(\mathcal{O}_t = 1 | s_t, a_t) = \exp(r(s_t, a_t)) \; .$$</span></p>
  <p>My question is why do they do this? In the paper they make no assumptions about the value of the rewards (e.g. bounding it to be non-positive) so in theory the rewards can take any value and thus the RHS can be larger than 1. This is obviously invalid for a probability. It would make sense if there was some normalising constant, or if the author said that the probability is proportional to this, but they don't.</p>
  <p>I have searched online and nobody seems to have asked this question which makes me feel like I am missing something quite obvious so I would appreciate if somebody could clear this up for me please.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-19T18:56:45.627Z
Comments.Text: Some papers that I've come across assumed that the reward is normalized to be in a certain range. Not sure if the author is _implicitly_ assuming that in that context or not. Alternatively, he may be using $\operatorname{exp}$ to denote any exponential function with codomain $[0, 1]$, for instance, the sigmoid, but that seems a bit uncommon, given that $\operatorname{exp}$ is typically used to denote $e$. My last guess is that the author was careless. I didn't really read the paper and these are more guesses. You could try to e-mail him.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-19T19:45:13.377Z
Comments.Text: @nbro I’d noticed it in another paper I’d read briefly but I can’t remember it now (it wasn’t really relevant for my research) so I can’t remember if they made the assumptions on the range of the rewards. I think I’ll try and email the author and maybe answer this myself.
Comments.UserDisplayName: David
Tags: <reinforcement-learning><probability><inference><probabilistic-graphical-models>
Title: In RL as probabilistic inference, why do we take a probability to be $\exp(r(s_t, a_t))$?
UserDisplayName: David
ThreadText: # In RL as probabilistic inference, why do we take a probability to be $\exp(r(s_t, a_t))$?
  
  (Asked by: username_0 on 2020-12-18)
  
  In section 2 the paper [Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review](https://arxiv.org/pdf/1805.00909.pdf) the author is discussing formulating the RL problem as a probabilistic graphical model. They introduce a binary optimality variable $\mathcal{O}\_t$ which denotes whether time step $t$ was optimal (1 if so, 0 otherwise). They then define the probability that this random variable equals 1 to be
  
  $$\mathbb{P}(\mathcal{O}\_t = 1 | s\_t, a\_t) = \exp(r(s\_t, a\_t)) \; .$$
  
  My question is why do they do this? In the paper they make no assumptions about the value of the rewards (e.g. bounding it to be non-positive) so in theory the rewards can take any value and thus the RHS can be larger than 1. This is obviously invalid for a probability. It would make sense if there was some normalising constant, or if the author said that the probability is proportional to this, but they don't.
  
  I have searched online and nobody seems to have asked this question which makes me feel like I am missing something quite obvious so I would appreciate if somebody could clear this up for me please.
  
  **Tags:** reinforcement-learning, probability, inference, probabilistic-graphical-models
  
  > **Comments:**
  > **username_1**: Some papers that I've come across assumed that the reward is normalized to be in a certain range. Not sure if the author is _implicitly_ assuming that in that context or not. Alternatively, he may be using $\operatorname{exp}$ to denote any exponential function with codomain $[0, 1]$, for instance, the sigmoid, but that seems a bit uncommon, given that $\operatorname{exp}$ is typically used to denote $e$. My last guess is that the author was careless. I didn't really read the paper and these are more guesses. You could try to e-mail him.  
  > **username_0**: @username_1 I’d noticed it in another paper I’d read briefly but I can’t remember it now (it wasn’t really relevant for my research) so I can’t remember if they made the assumptions on the range of the rewards. I think I’ll try and email the author and maybe answer this myself.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2020-12-23. Score: 0 | Accepted answer
  
  After doing some further reading, it turns out that negative rewards *are* an assumption for this distribution to hold. However, the author notes that as long as you don't receive a reward of infinity for any action then it is possible to re-scale your rewards by subtracting the maximum value of your potential rewards so that they are always negative.
  
  
  ---
  
  