Answers.Body: <p>I think your equations are alright.</p>
  <p>Anyway, this is just a question of mathematical notation.</p>
  <p>In measure theory, a discrete random variable <span class="math-container">$X$</span> is said to have a counting measure over it's support <span class="math-container">$\mathcal X = \{x_k,k=1,...K\}$</span>.</p>
  <p>We can define its distribution function as <span class="math-container">$F_X(x)=\sum_k p(x_k) 1(x \ge x_k)$</span>.</p>
  <p>With the distribution function, we can write the expectation as the measure-theoretical integration <span class="math-container">$EX=\int _\mathcal X xF_X(dx)$</span>.</p>
  <p>Notice the part <span class="math-container">$F_X(dx)$</span>, which is not the usual <span class="math-container">$dF_X(x)$</span> notation for Riemann integration.</p>
  <p>This is why many graduate-level texts prefer to measure theory to define integration and expectations - so that we can use one unified notation for discrete random variables and continuous random variables, and any random variable you can think of living in some abstract measure space <span class="math-container">$(\Omega, \mathcal F, \mathbb P)$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-13T06:37:11.663Z
Answers.LastActivityDate: 2022-08-13T06:37:11.663Z
Answers.UserDisplayName: easyliving
Body: <p>I'm currently studying Reinforcement Learning and I'd like to know what would be the Bellman optimality equation for action values <span class="math-container">$q_∗(s, a)$</span> for a MDP with continuous states and actions, written out using explicit integration (no expectation notation).</p>
  <p>The discrete case is</p>
  <p><span class="math-container">$$
  q_*(s,a)  = \sum_{s'}\sum_r p(s',r|s,a)[r+\gamma \max_{a'}q*(s',a')]
  $$</span></p>
  <p>My thoughts for the continuous case are:</p>
  <p><span class="math-container">\begin{align}
  q_*(s,a) &amp;= \int_{s'}\int_{r}f_{s',r|s,a}(s',r|s,a)[r+\gamma\max_{a'}q_*(s',a')]drds'
  \end{align}</span></p>
  <p>Is this how it would look like?</p>
  
Tags: <reinforcement-learning><bellman-equations><continuous-action-spaces><continuous-state-spaces>
Title: What would be the Bellman optimality equation for $q_∗(s, a)$ for an MDP with continuous states and actions?
UserDisplayName: user
ThreadText: # What would be the Bellman optimality equation for $q_∗(s, a)$ for an MDP with continuous states and actions?
  
  (Asked by: username_0 on 2022-08-12)
  
  I'm currently studying Reinforcement Learning and I'd like to know what would be the Bellman optimality equation for action values $q\_∗(s, a)$ for a MDP with continuous states and actions, written out using explicit integration (no expectation notation).
  
  The discrete case is
  
  $$
  q\_\*(s,a) = \sum\_{s'}\sum\_r p(s',r|s,a)[r+\gamma \max\_{a'}q\*(s',a')]
  $$
  
  My thoughts for the continuous case are:
  
  \begin{align}
  q\_\*(s,a) &= \int\_{s'}\int\_{r}f\_{s',r|s,a}(s',r|s,a)[r+\gamma\max\_{a'}q\_\*(s',a')]drds'
  \end{align}
  
  Is this how it would look like?
  
  **Tags:** reinforcement-learning, bellman-equations, continuous-action-spaces, continuous-state-spaces
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-08-13. Score: 1 | Accepted answer
  
  I think your equations are alright.
  
  Anyway, this is just a question of mathematical notation.
  
  In measure theory, a discrete random variable $X$ is said to have a counting measure over it's support $\mathcal X = \{x\_k,k=1,...K\}$.
  
  We can define its distribution function as $F\_X(x)=\sum\_k p(x\_k) 1(x \ge x\_k)$.
  
  With the distribution function, we can write the expectation as the measure-theoretical integration $EX=\int \_\mathcal X xF\_X(dx)$.
  
  Notice the part $F\_X(dx)$, which is not the usual $dF\_X(x)$ notation for Riemann integration.
  
  This is why many graduate-level texts prefer to measure theory to define integration and expectations - so that we can use one unified notation for discrete random variables and continuous random variables, and any random variable you can think of living in some abstract measure space $(\Omega, \mathcal F, \mathbb P)$.
  
  
  ---
  
  