Why does conditional expectation have this property for independent random variables?
For a reference, please see pp. 53-54 of Boucheron, Lugosi, Massart, Concentration Inequalities: A Nonasymptotic Theory of Independence. Let $f: \mathcal{X}^n \to \mathbb{R}$ be a measurable function ($\mathcal{X}$ is some measurable space), and let $X_1, \dots, X_n$ be independent random variables taking values in $\mathcal{X}$.
Claim: Because of Fubini's theorem, one can write that $$\mathbb{E}[f(X_1, \dots, X_n)| X_1, \dots, X_i] = \int\limits_{\mathcal{X}^{n-i}} f(X_1, \dots, X_i, x_{i+1}, \dots, x_n) d\mu_{i+1}(x_{i+1}) \dots d\mu_n(x_n) $$ where for each $i$, $\mu_i$ denotes the probability distribution of $X_i$. Analogously, $$\mathbb{E}[f(X_1, \dots, X_n)|X_1, \dots, X_{i-1}, X_{i+1}, \dots, X_n] = \int\limits_{\mathcal{X}} f(X_1, \dots, X_{i-1}, x_i, X_{i+1}, \dots, X_n)d\mu_i(x_i) $$
This claim is central to the argument used to show the Efron-Stein inequality in the book mentioned above. Analogous claims were also used extensively in several book excerpts about U-statistics which I was reading a couple of months ago, but by now I have forgotten the specific references.
Anyway, this appears to be an important computational property of conditional expectation, and one that has at least two applications in statistics. It seems plausible to me, but even after figuring out what I need to verify, it still doesn't seem like the integrals involved are well-defined.
Question: Why are the equations below well-defined? In particular, how can the left- and right-hand sides be equal if we integrate over some coordinates twice on the left-hand side but integrate over those same coordinates only once on the right-hand side?
For the first formula:
It suffices to show that, for all $S_1 \in \sigma(X_1), \dots, S_i \in \sigma(X_i)$ : $$\int\limits_{S_1 \times \dots \times S_i \times \mathcal{X} \times \dots \times \mathcal{X} } \left( \int_{\mathcal{X}^{n-i}} f(X_1, \dots, X_i, x_{i+1}, \dots, x_n) d\mu_{i+1}(x_{i+1}) \dots d\mu_n(x_n) \right) d\nu(x_1, \dots, x_n) =$$ $$ \int\limits_{S_1 \times \dots \times S_i \times \mathcal{X} \times \dots \times \mathcal{X} } f(X_1, \dots, X_n) d\nu(x_1, \dots, x_n) \,,$$ where $\nu$ is meant to denote the joint probability measure on $\mathcal{X}^n$, which is the product measure of all of the $\mu_j$'s because the $X_j$ are independent.
For the second formula:
It suffices to show for all $S_1 \in \sigma(X_1), \dots, S_{i-1} \in \sigma(X_{i-1}), S_{i+1} \in \sigma(X_{i+1}), \dots, S_n \in \sigma(X_n)$ that:
$$\int\limits_{S_1 \times \dots \times S_{i-1} \times \mathcal{X} \times S_{i+1} \times \dots \times S_n} \left( \int\limits_{\mathcal{X}} f(X_1, \dots, X_{i-1}, x_i, X_{i+1}, \dots, X_n) d\mu_i(x_i) \right) d\nu (x_1, \dots, x_n) =$$ $$ \int\limits_{S_1 \times \dots \times S_{i-1} \times \mathcal{X} \times S_{i+1} \times \dots \times S_n } f(X_1, \dots, X_n) d\nu(x_1, \dots, x_n) \,. $$
mathematical-statistics independence conditional-expectation probability-inequalities integral
Chill2Macht
Chill2MachtChill2Macht
Attempted answer: Fubini's theorem definitely only applies for the product measure, see here. So this is why we definitely need independence in order to be able to use Fubini's theorem in any way.
If we denote the $\sigma$-algebra assumed to exist on $\mathcal{X}$ by $\mathscr{F}$, then it should follow that the $\sigma$-algebra $$\sigma\left(\underbrace{\mathscr{F} \times \dots \times \mathscr{F}}_{n\ \text{times}}\right)=:\sigma(\mathscr{F}^n) \,,$$ the $\sigma$-algebra generated by the $n$-fold (set-wise) Cartesian product of $\mathscr{F}$ (i.e. by sets of the form $F_1 \times \dots \times F_n$ where for each $j$, $F_j \in \mathscr{F}$), is the implied $\sigma$-algebra on $\mathcal{X}^n$.
Write $\mathscr{N}$ (for null) for the trivial $\sigma$-algebra on $\mathcal{X}$, i.e. $\mathscr{N} = \{ \emptyset, \mathcal{X} \}$.
Originally, in a lot of places below I was using the full $\sigma$-algebra, $\mathscr{F}$, when I should instead have been using $\mathscr{N}$, which lead to much of my confusion. One reason why using $\mathscr{F}$ instead of $\mathscr{N}$ is wrong is the following: $\sigma(X_1), \sigma(X_1, X_2), \dots, \sigma(X_1, \dots, X_n)$ is an increasing filtration, i.e. each $\sigma$-algebra contains "more" sets than the $\sigma$-algebra preceding it in the filtration. This follows by using $\mathscr{N}$ where it is used in the places below, but if I had used $\mathscr{F}$ instead, then it would have been a decreasing filtration, which is completely incorrect. For another reason, we want, e.g., that $\sigma(X_1,X_2)$ is "isomorphic" to a $\sigma$-algebra on $\mathcal{X}^2$, which would only make sense/be possible if the "information" on the remaining coordinates was negligible, i.e. could be completely ignored. That follows if one uses $\mathcal{N}$, but the exact opposite is true if one uses $\mathscr{F}$ - in fact, in that case, one has more "information" about the remaining coordinates than coordinates 1 and 2, not less, and certainly not no information.
Moreover, we have that for each $j$, $\sigma(X_j) \subset \mathscr{F}$. So it makes sense to say that the $\sigma$-algebra $\sigma(X_1, \dots, X_i)$, corresponding to conditioning by $X_1, \dots, X_i$, is equal to $$\sigma\left(\sigma(X_1)\times \dots \times \sigma(X_i) \times \underbrace{\mathscr{N} \times \dots \times \mathscr{N}}_{n-i\ \text{times}}\right) =: \sigma((X_1,\dots, X_i) \times \mathscr{N}^{n-i}) \,, $$ the $\sigma$-algebra generated by sets of the form $$S_1 \times \dots \times S_i \times \underbrace{\mathcal{X} \times \dots \times \mathcal{X}}_{n-i\ \text{times}} \,,$$ where for each $1 \le j \le i$, $S_j \in \sigma(X_j)$ (the event $S_j$ is measurable with respect to the $\sigma$-field generated by $X_j$). (Note that the Cartesian of any product with the empty set is again the empty set, hence why we don't also need to add sets where at least one of the $j$th coordinates, for $j > i$, is the empty set, because all such sets are the empty set, and the empty set is in every $\sigma$-algebra.)
Likewise, the $\sigma$-algebra $\sigma(X_1, \dots, X_{i-1}, X_{i+1}, \dots, X_n)$, corresponding to conditioning on every random variable besides $X_i$, is equal to $$\sigma\left( \sigma(X_1) \times \dots \times \sigma(X_{i-1}) \times \mathscr{N} \times \sigma(X_{i+1}) \times \dots \times \sigma(X_n) \right) =: \sigma( (X_1, \dots, X_{i-1}) \times \mathscr{N} \times (X_{i+1} \times X_n)) \,, $$ the $\sigma$-algebra generated by sets of the form $S_1 \times \dots \times S_{i-1} \times \mathcal{X} \times S_{i+1} \times \dots \times S_n$, where for each $j\not=i$, $S_j$ is measurable with respect to the $\sigma$-algebra induced by $X_j$.
This serves at least two points:
To clarify that even without assuming independence of the $X_j$, i.e. without assuming that the joint probability measure on $\mathcal{X}^n$ is equal to the product of the marginal probability measures on each copy of $\mathcal{X}$, we should still have that the $\sigma$-algebra on $\mathcal{X}^n$ is (generated by) the product of the $\sigma$-algebra's on each of the respective copies of $\mathcal{X}$.
Showing that the measurability criteria corresponding to our conditional expectation operators can be decomposed into simpler component-wise criteria. What I mean by this is that, to check that a random variable is measurable w.r.t. a $\sigma$-algebra generated by some class of sets, it suffices to show that it is measurable with respect to the generating class. So even if arbitrary sets in the $\sigma$-algebras on $\mathcal{X}^n$ in question can have a very complicated form, the generating sets for these $\sigma$-algebras are all fairly simple, at least inasmuch as they are no more complicated than any of the $\mathscr{F}, \sigma(X_1), \dots, \sigma(X_n)$ component-wise.
This reduced class of sets to check works not only for checking measurability, but also for checking the definition of conditional expectation (I think).
I have to look up the precise theorem statement in Dudley's Real Analysis and Probability (assuming such a theorem exists) that shows that checking measurability on the generating sets is sufficient, but I have to imagine that the proof is entirely analogous to the proof that it is sufficient to check continuity by checking the pre-images of bases which generate that topology (e.g. open intervals in the case of the usual topology on $\mathbb{R}$).
Therefore an outline of a proof could be as follows:
To show that the right-hand side satisfies the definition of conditional expectation, it suffices to show that, for all $S_1 \in \sigma(X_1), \dots, S_i \in \sigma(X_i)$ : $$\int\limits_{S_1 \times \dots \times S_i \times \mathcal{X} \times \dots \times \mathcal{X} } \left( \int_{\mathcal{X}^{n-i}} f(X_1, \dots, X_i, x_{i+1}, \dots, x_n) d\mu_{i+1}(x_{i+1}) \dots d\mu_n(x_n) \right) d\nu(x_1, \dots, x_n) = \int\limits_{S_1 \times \dots \times S_i \times \mathcal{X} \times \dots \times \mathcal{X} } f(X_1, \dots, X_n) d\nu(x_1, \dots, x_n) \,,$$ where $\nu$ is meant to denote the joint probability measure on $\mathcal{X}^n$.
It kind of looks like the integral as written on the left-hand might not be well-defined. I still need to think about this more.
Second formula:
To show that the right-hand side satisfies the definition of conditional expectation, it should be sufficient to show that, for all $S_1 \in \sigma(X_1), \dots, S_{i-1} \in \sigma(X_{i-1}), S_{i+1} \in \sigma(X_{i+1}), \dots, S_n \in \sigma(X_n)$:
$$\int\limits_{S_1 \times \dots \times S_{i-1} \times \mathcal{X} \times S_{i+1} \times \dots \times S_n} \left( \int\limits_{\mathcal{X}} f(X_1, \dots, X_{i-1}, x_i, X_{i+1}, \dots, X_n) d\mu_i(x_i) \right) d\nu (x_1, \dots, x_n) = \int\limits_{S_1 \times \dots \times S_{i-1} \times \mathcal{X} \times S_{i+1} \times \dots \times S_n } f(X_1, \dots, X_n) d\nu(x_1, \dots, x_n) \,. $$
3 revs
Not the answer you're looking for? Browse other questions tagged mathematical-statistics independence conditional-expectation probability-inequalities integral or ask your own question.
Conditional expectation of random vector given low-rank linear transform
Find conditional expectation given a discrete random variable whose range is N
Proving an identity involving sample variance
How to improve an estimator for a Poisson sample
If $X_1,\cdots,X_n \sim \mathcal{N}(\mu, 1)$ are IID, then compute $\mathbb{E}\left( X_1 \mid T \right)$, where $T = \sum_i X_i$