Answers.Body: <p>The confusing part is how the variance is defined in terms of <span class="math-container">$P^{\pi}Z(x, a)$</span>, which includes three sources of randomness, <span class="math-container">$X', A', Z$</span>. However, if we define the variance as</p>
  <p><span class="math-container">\begin{align*}
  \mathbb{V}(P^{\pi}Z(x, a)) &amp; \triangleq \int P(X' = x', A' = a') \mathbb{V} (Z | x', a')dx'da' \\
   &amp; = \mathbb{E}_{X', A'}[\mathbb{V}(Z|X', A')]
  \end{align*}</span></p>
  <p>Then the result clearly follows as both <span class="math-container">$P^{\pi}Z_{1}$</span> and <span class="math-container">$P^{\pi}Z_{2}$</span> evolve the same dynamics, that is, on <span class="math-container">$X', A'$</span>. By linearity of the expectation, the statement concludes.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-01T13:21:46.410Z
Answers.Text: Your notation of the variance is quite strange on the right hand-side. $Z$ is a function with the arguments $(x, a)$, it is not an argument to the probability $\mathbb{V}$, like it looks in your notation. Also you somehow define the variance as the expectation value over exactly the same variance, could you somehow distinguish these obviously different variances from each other?
Answers.UserDisplayName: pythonic833
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-01T13:35:41.483Z
Answers.Text: Firstly, $Z$ is a random variable, whose distribution describes the possible values of the action-value function. And you can think of the expression as that the left-hand side variance is defined as the expected value of conditional variance of $Z$, because it is natural to have distribution of $Z$ after the next state and action are ready.
Answers.UserDisplayName: Magi Feeney
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-01T12:18:25.157Z
Answers.LastActivityDate: 2023-03-01T12:18:25.157Z
Answers.UserDisplayName: Magi Feeney
Answers.Body: <p>This is a common trick in reinforcement learning literature which uses the law of large numbers to use the sampled variables <span class="math-container">$X$</span> and <span class="math-container">$A$</span> instead of <span class="math-container">$x$</span> and <span class="math-container">$a$</span>. Let's say we know the probability <span class="math-container">$p(x)$</span> with which a continuous <span class="math-container">$x$</span> is given, then we can calculate the expectation value of <span class="math-container">$x$</span> as:
  <span class="math-container">\begin{align}
  \mathbb{E}_{x \sim p}[x] &amp;= \int p(x)x ~dx \\
                &amp;= \lim \limits_{N \to \infty}\frac{1}{N}\sum_{N} X
  \end{align}</span>
  In the last line, we did not assume any knowledge of <span class="math-container">$p(x)$</span> but we averaged over the samples taken from <span class="math-container">$p(x)$</span>. This would be the only way to find the expectancy value for an unknown system, e.g. the average speed of a driver, of whom we don't know the exact driving behaviour.
  We can also use the sampled variables for an expression of the variance
  <span class="math-container">\begin{align}
  \mathbb{V}(x) = \int p(x)(x - \mathbb{E}_{x \sim p}(x))² dx  = \lim \limits_{N \to \infty} \frac{1}{N} \sum_{N} (X-\mathbb{E}[X])²
  \end{align}</span>
  From here, two things could happen, since the notation in the paper is not quite clear.</p>
  <ol>
  <li>Define <span class="math-container">$\mathbb{V}[X] = (X-\mathbb{E}[X])²$</span> then
  <span class="math-container">\begin{equation}
  \lim \limits_{N \to \infty} \frac{1}{N} \sum_{N} (X-\mathbb{E}[X])² = \lim \limits_{N \to \infty} \frac{1}{N} \sum_{N} \mathbb{V}[X] = \mathbb{E}[{\mathbb{V}[X]}].
  \end{equation}</span>
  But this would be a rather confusing notation since <span class="math-container">$(X-\mathbb{E}[X])²$</span> is only sample whose expecation value is the variance.</li>
  <li>Since the variance already is an expectation value (the expected, absolute difference from the mean), we can add an additional expectation value, which basically does not do anything:
  <span class="math-container">\begin{equation}
  \lim \limits_{N \to \infty} \frac{1}{N} \sum_{N} (X-\mathbb{E}[X])² = \mathbb{V}[X] = \mathbb{E}[\mathbb{V}[X]] = \lim \limits_{N \to \infty} \frac{1}{N} \sum_{N} \mathbb{V}[X] = \lim \limits_{N \to \infty} \frac{1}{N} N\mathbb{V}[X] = \mathbb{V}[X].
  \end{equation}</span>
  I lean towards this explanation, but still introducing a useless expectation value is quite strange.</li>
  </ol>
  <p>Now, let's use this in order to explain why:
  <span class="math-container">\begin{align}
  \mathbb{V}(P^{\pi}Z_{1}(x, a)) - \mathbb{V}(P^{\pi}Z_{2}(x, a)) \\
   = \int \pi(a) p(x)(P^{\pi}Z_{1}(x, a) - \mathbb{E}_{x \sim p, a\sim \pi}[{P^{\pi}Z_{1}(x, a)}])^{2}~dxda 
  - \int \pi(a) p(x)(P^{\pi}Z_{2}(x, a) - \mathbb{E}_{x \sim p, a\sim \pi}[{P^{\pi}Z_{2}(x, a)}])^{2}~dxda \\
  = \lim \limits_{N \to \infty} \frac{1}{N} \sum (Z_{1}(X', A') - \mathbb{E}[Z_{1}(X', A')])^{2} - (Z_{2}(X', A') - \mathbb{E}[Z_{2}(X', A')])^{2} \\
  = \mathbb{E}[\mathbb{V}(Z_{1}(X', A')) - \mathbb{V}(Z_{2}(X', A'))] \\
  \end{align}</span>
  Where in the last step one can use any of the two possibilities given above. I abbreviated <span class="math-container">$p(x') = p(x'|x, a)$</span> and <span class="math-container">$\pi(a) = \pi(a|x)$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-01T10:22:58.697Z
Answers.Text: You are misleading two things, 1) x, a is conditional-ed, no randomness; 2) variance definition is wrong.
Answers.UserDisplayName: Magi Feeney
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-01T10:43:17.633Z
Answers.Text: Oh, I forgot the squares, you are right. I also just used a short-hand notation and therefore dropped the conditionals. But the randomness comes directly into play when capital letters are used, e.g. see from the paper 1. Introduction
  > "Bellman’s equation succintly
  describes this value in terms of the expected reward and ex-
  pected outcome of the random transition $(x, a) → (X′, A′)$."
  So,
  $(x, a)$ is known but due to a random transition we end up in the random $(X', A')$
Answers.UserDisplayName: pythonic833
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-01T12:08:02.410Z
Answers.Text: I think it is inappropriate to integrate over (x, a), this is the definition of the expectation which variable has randomness. It is crucial to determine how the variance of $P^{\pi}Z(x, a)$ is defined. I will update an answer in a later time.
Answers.UserDisplayName: Magi Feeney
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-01T08:35:35.950Z
Answers.LastActivityDate: 2023-03-01T10:33:49.267Z
Answers.LastEditDate: 2023-03-01T10:33:49.267Z
Answers.UserDisplayName: pythonic833
Body: <p>I am stuck at the proof of the contraction of variance for distributional Bellman operator from the <a href="https://arxiv.org/pdf/1707.06887.pdf" rel="nofollow noreferrer">paper</a>, in which it is defined as
  <a href="https://i.sstatic.net/iTUgW.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/iTUgW.png" alt="enter image description here" /></a></p>
  <p>and the proof is stated as</p>
  <p><a href="https://i.sstatic.net/t9CMK.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/t9CMK.png" alt="enter image description here" /></a></p>
  <p>In its second part, how is the  variance of the target distribution equal to the expectation of the  variance terms over the next state-action pairs?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-01-30T14:35:10.067Z
Comments.Text: Do you mean why $\mathbb{V}(P^\pi Z_1(x, a)) - \mathbb{V}(P^\pi Z_2x, a))$ turns into $\mathbb{E}[\mathbb{V}(Z_1(X', A')) - \mathbb{V}(Z_2(X', A'))]$?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-01-30T15:48:42.723Z
Comments.Text: @nbro Yes, exactly.
Comments.UserDisplayName: Magi Feeney
Tags: <reinforcement-learning><bellman-equations><conditional-probability><bellman-operators>
Title: How the proof of the contraction of variance for distributional Bellman operator follows
UserDisplayName: Magi Feeney
ThreadText: # How the proof of the contraction of variance for distributional Bellman operator follows
  
  (Asked by: username_0 on 2023-01-30)
  
  I am stuck at the proof of the contraction of variance for distributional Bellman operator from the [paper](https://arxiv.org/pdf/1707.06887.pdf), in which it is defined as
  [![enter image description here](https://i.sstatic.net/iTUgW.png)](https://i.sstatic.net/iTUgW.png)
  
  and the proof is stated as
  
  [![enter image description here](https://i.sstatic.net/t9CMK.png)](https://i.sstatic.net/t9CMK.png)
  
  In its second part, how is the variance of the target distribution equal to the expectation of the variance terms over the next state-action pairs?
  
  **Tags:** reinforcement-learning, bellman-equations, conditional-probability, bellman-operators
  > **Comments:**
  > **username_1**: Do you mean why $\mathbb{V}(P^\pi Z_1(x, a)) - \mathbb{V}(P^\pi Z_2x, a))$ turns into $\mathbb{E}[\mathbb{V}(Z_1(X', A')) - \mathbb{V}(Z_2(X', A'))]$?  
  > **username_0**: @nbro Yes, exactly.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_0 on 2023-03-01. Score: 0
  
  The confusing part is how the variance is defined in terms of $P^{\pi}Z(x, a)$, which includes three sources of randomness, $X', A', Z$. However, if we define the variance as
  
  \begin{align\*}
  \mathbb{V}(P^{\pi}Z(x, a)) & \triangleq \int P(X' = x', A' = a') \mathbb{V} (Z | x', a')dx'da' \\
  & = \mathbb{E}\_{X', A'}[\mathbb{V}(Z|X', A')]
  \end{align\*}
  
  Then the result clearly follows as both $P^{\pi}Z\_{1}$ and $P^{\pi}Z\_{2}$ evolve the same dynamics, that is, on $X', A'$. By linearity of the expectation, the statement concludes.
  
  > **Comments:**
  > **username_2**: Your notation of the variance is quite strange on the right hand-side. $Z$ is a function with the arguments $(x, a)$, it is not an argument to the probability $\mathbb{V}$, like it looks in your notation. Also you somehow define the variance as the expectation value over exactly the same variance, could you somehow distinguish these obviously different variances from each other?  
  > **username_0**: Firstly, $Z$ is a random variable, whose distribution describes the possible values of the action-value function. And you can think of the expression as that the left-hand side variance is defined as the expected value of conditional variance of $Z$, because it is natural to have distribution of $Z$ after the next state and action are ready.  
  
  ---
  
  ### Answer by username_2 on 2023-03-01. Score: 0
  
  This is a common trick in reinforcement learning literature which uses the law of large numbers to use the sampled variables $X$ and $A$ instead of $x$ and $a$. Let's say we know the probability $p(x)$ with which a continuous $x$ is given, then we can calculate the expectation value of $x$ as:
  \begin{align}
  \mathbb{E}\_{x \sim p}[x] &= \int p(x)x ~dx \\
  &= \lim \limits\_{N \to \infty}\frac{1}{N}\sum\_{N} X
  \end{align}
  In the last line, we did not assume any knowledge of $p(x)$ but we averaged over the samples taken from $p(x)$. This would be the only way to find the expectancy value for an unknown system, e.g. the average speed of a driver, of whom we don't know the exact driving behaviour.
  We can also use the sampled variables for an expression of the variance
  \begin{align}
  \mathbb{V}(x) = \int p(x)(x - \mathbb{E}\_{x \sim p}(x))² dx = \lim \limits\_{N \to \infty} \frac{1}{N} \sum\_{N} (X-\mathbb{E}[X])²
  \end{align}
  From here, two things could happen, since the notation in the paper is not quite clear.
  
  1. Define $\mathbb{V}[X] = (X-\mathbb{E}[X])²$ then
     \begin{equation}
     \lim \limits\_{N \to \infty} \frac{1}{N} \sum\_{N} (X-\mathbb{E}[X])² = \lim \limits\_{N \to \infty} \frac{1}{N} \sum\_{N} \mathbb{V}[X] = \mathbb{E}[{\mathbb{V}[X]}].
     \end{equation}
     But this would be a rather confusing notation since $(X-\mathbb{E}[X])²$ is only sample whose expecation value is the variance.
  2. Since the variance already is an expectation value (the expected, absolute difference from the mean), we can add an additional expectation value, which basically does not do anything:
     \begin{equation}
     \lim \limits\_{N \to \infty} \frac{1}{N} \sum\_{N} (X-\mathbb{E}[X])² = \mathbb{V}[X] = \mathbb{E}[\mathbb{V}[X]] = \lim \limits\_{N \to \infty} \frac{1}{N} \sum\_{N} \mathbb{V}[X] = \lim \limits\_{N \to \infty} \frac{1}{N} N\mathbb{V}[X] = \mathbb{V}[X].
     \end{equation}
     I lean towards this explanation, but still introducing a useless expectation value is quite strange.
  
  Now, let's use this in order to explain why:
  \begin{align}
  \mathbb{V}(P^{\pi}Z\_{1}(x, a)) - \mathbb{V}(P^{\pi}Z\_{2}(x, a)) \\
  = \int \pi(a) p(x)(P^{\pi}Z\_{1}(x, a) - \mathbb{E}\_{x \sim p, a\sim \pi}[{P^{\pi}Z\_{1}(x, a)}])^{2}~dxda
  - \int \pi(a) p(x)(P^{\pi}Z\_{2}(x, a) - \mathbb{E}\_{x \sim p, a\sim \pi}[{P^{\pi}Z\_{2}(x, a)}])^{2}~dxda \\
  = \lim \limits\_{N \to \infty} \frac{1}{N} \sum (Z\_{1}(X', A') - \mathbb{E}[Z\_{1}(X', A')])^{2} - (Z\_{2}(X', A') - \mathbb{E}[Z\_{2}(X', A')])^{2} \\
  = \mathbb{E}[\mathbb{V}(Z\_{1}(X', A')) - \mathbb{V}(Z\_{2}(X', A'))] \\
  \end{align}
  Where in the last step one can use any of the two possibilities given above. I abbreviated $p(x') = p(x'|x, a)$ and $\pi(a) = \pi(a|x)$.
  
  > **Comments:**
  > **username_0**: You are misleading two things, 1) x, a is conditional-ed, no randomness; 2) variance definition is wrong.  
  > **username_2**: Oh, I forgot the squares, you are right. I also just used a short-hand notation and therefore dropped the conditionals. But the randomness comes directly into play when capital letters are used, e.g. see from the paper 1. Introduction > "Bellman’s equation succintly describes this value in terms of the expected reward and ex- pected outcome of the random transition $(x, a) → (X′, A′)$." So, $(x, a)$ is known but due to a random transition we end up in the random $(X', A')$  
  > **username_0**: I think it is inappropriate to integrate over (x, a), this is the definition of the expectation which variable has randomness. It is crucial to determine how the variance of $P^{\pi}Z(x, a)$ is defined. I will update an answer in a later time.  
  
  ---
  
  