Regular conditional distribution vs conditional distribution What is the difference between the concept of the regular conditional distribution and the concept of the conditional distribution? 
Why do we need these two different concepts?
Under which circumstances to they coincide?
Is it true that the regular conditional distribution is in general not a probability measure? If so why and why would we use it then?
I think a somewhat related question is:
What is the difference between the elementary conditional probability $\mathbb{P}(A|B)=\frac{\mathbb{P}(A,B)}{\mathbb{P}(B)}$ for sets $A, B \in \mathcal{A}$ and the (general) conditional probability defined via the conditional expectation: $\mathbb{P}(X|Y)(B)=\mathbb{E}[\mathbb{I}_{X^{-1}(B)}|Y]$?


Why do we need these two different concepts?

Regular conditional distributions are useful because they allow us to generalize the elementary notions of conditional distribution where we consider ratios of the form $\frac{\Pr[A\cap B]}{\Pr[A]}$. This formalization is in contrast with the Kolmogorov abstract conditional expectation which is up to a.s. equivalence. There are also some useful applications such as allowing us to evaluate conditional expectations as ordinary expectations.

Under which circumstances do they coincide?

Let $(\Omega, \mathcal{F}, \Pr)$ be a probability space.
Recall, the elementary definition,
$$\Pr[A|X=x]=\frac{\Pr[\{X=x\} \cap A]}{\Pr[X=x]}$$
This definition gives us a probability measure on the space that concentrates on the event $\{X=x\}$. Since $\{X\neq x\}\cap\{X=x\}=\emptyset$. And for any $A\in\mathcal{F}$ we have,
$$\Pr[A]=\sum_{x\in\mathbb{R}}\Pr[X=x]\Pr[A|X=x]$$
Equivalently,
$$\mathbb{E}[A]=\sum_{x\in\mathbb{R}}\mathbb{E}[A|X=x]\Pr[X=x]$$
The issue that we run into here is that Kolmogorov's abstract conditional expectation is arbitrary on any event with measure zero and it could happen that stringing together uncountably many events of measure zero yields an event with positive probability. This serves as motivation for introducing regular conditional distributions. The regular conditional distribution is a disintegration that allows us to avoid problems with uncountably many measure-zero sets. The other benefit is the disintegration formula which allows us to think of conditional expectations as just regular expectations taken with respect to a conditional measure. That is, let $\mu_\omega(dx)$ be a regular conditional distribution for $X$ and given a sub-$\sigma$-algebra $\mathcal{G}\subset\mathcal{F}$ with r.v. $Y$ that is $\mathcal{G}$-measurable, and a jointly measurable and integrable $f(X,Y)$ we have,
$$\mathbb{E}[f(X,Y)|\mathcal{G}]=\int f(x,Y(\omega))\mu_\omega(dx)$$
holds almost surely. This is very useful as it allows us to pass to a regular expectation. A classic example is the proof of the conditional Jensen inequality (see Dudley, Real Analysis and Probability, 2004 Section 10.2)
The downside to using regular conditional distributions is that we need slightly stronger assumptions than those used by the Kolmogorov abstract conditional expectation for existence. The proof is more technical also and requires that we have a Hausdorff topological space (Theorem 10.2.2 in Dudley). In particular, for the most general existence theorem, we need that our random variable maps into a Polish space.

Is it true that the regular conditional distribution is in general not a probability measure?

In fact, a regular conditional distribution is by definition a probability measure on the range of the random variable. Let $X$ be a random variable from our probability space into $(T, \mathcal{T})$. Let $\mathcal{G}$ be a sub-$\sigma$-algebra as above.
Recall the definition of a regular conditional probability is a function $P(\cdot|\mathcal{G})(\cdot)$ on $\mathcal{F}\times\Omega$ so that,

*

*For $B\in\mathcal{F}$,

$$P(B|\mathcal{G})(\cdot)=\Pr[X\in B|\mathcal{G}](\cdot)$$
almost surely where $P(B|\mathcal{G})(\cdot)$ is a $\mathcal{G}$-measurable.


*$P(\cdot|\mathcal{G})(\omega)$ is a probability measure on $\mathcal{F}$.

The regular conditional distribution of $X$ given $\mathcal{G}$ is the function $\mu_\omega(\cdot):\Omega\times\mathcal{T}\to[0,1]$ such that,

*

*$\mu_\omega(\cdot)$ is a probability measure on $\mathcal{T}$ for almost all $\omega$.


*For $A\in\mathcal{T}$ we have,
$$\mu_\cdot(A)=\Pr[X^{-1}(A)|\mathcal{G}](\cdot)$$
where $\mu_\cdot(A)$ is $\mathcal{G}$-measurable.
Both objects can be thought of as Markov kernels (i.e. they are regular) and there is a very clear correspondence between the two concepts here because if we set $(T,\mathcal{T})=(\Omega,\mathcal{F})$ and let $X$ be the identity map then the regular conditional distribution is just the regular conditional probability. Since we just get that,
$$\mu_\omega(B)=\Pr[X^{-1}(B)|\mathcal{G}](\omega)=\Pr[B|\mathcal{G}](\omega)=P(B|\mathcal{G})(\omega)$$
Now defining a probability measure on our original space further underscoring the point. Essentially, what we are seeing is rather intuitive. The regular conditional probability acts as a conditional analog for the typical probability measure assigning probabilities to events in our sample space while the regular conditional distribution acts as the conditional analog to the distribution of our random variable i.e. a pushforward measure.
Finally, if we wanted to ignore regularity we might just try to define the conditional probability in terms of Kolmogorov's conditional expectation. That is,
$$\Pr[A|\mathcal{G}]=\mathbb{E}[1_A|\mathcal{G}]$$
In fact, this is a vector measure as it will assign the null set measure zero and it is countably additive ($\Pr[\cup_{n\in\mathbb{N}} A_n|\mathcal{G}]=\sum_{n\in\mathbb{N}}\Pr[A_n|\mathcal{G}]$ a.s.). But, it is not a probability measure because this definition is only up to a.s. equivalence and as we discussed earlier this means that when we consider uncountable families of measure-zero sets we may run into trouble. However, regularity will let us take care of this although as we have seen we need stronger assumptions.
Useful References:
Chang and Pollard, 1997 This is a highly recommended read as it is quite short and easy to read but contains many examples of why we should care about this concept given the abstract conditional expectation. The only downside is the notation is a bit unusual.
Pollard, 2001, Chapter 5
Dudley, 2004, Chapter 10
Addendum:
For completeness let us define Kolmogorov's abstract conditional expectation. It is enough to give the conditional expectation of $X$ given a sub-$\sigma$-algebra $\mathcal{G}\subset\mathcal{F}$. Then, for any $X\in L^1(\Omega, \mathcal{F},\Pr)$ the conditional expectation, $\mathbb{E}[X|\mathcal{G}]$, is the unique random variable $Z\in L^1(\Omega, \mathcal{F},\Pr)$ such that for all $G\in\mathcal{G}$,
$$\int_A Z d\Pr = \int_A X d\Pr$$
It suffices to consider this definition because we can always consider the case where this $\mathcal{G}$ is generated by a random variable to retrieve the elementary definition of conditional expectation. Also, take note that this definition only holds up to a.s. equality defined by an equivalence class of integrable functions. Thus, the conditional expectation can be defined by any member of that equivalence class. Thus, whenever we use this conditional expectation we have to understand that our reasoning only holds up to a.s. equality.
Typically, this definition is first stated for random variables $X\in L^2(\Omega,\mathcal{F},\Pr)$ where it coincides with the minimizer of $E[(Z-X)^2]$ and then generalize to $X\in L^1(\Omega, \mathcal{F},\Pr)$ so that we may account for non-square integrable random variables. It turns out this condition is also sufficient for the existence of the conditional expectation. The proof is a rather straightforward application of the Radon-Nikodym theorem (see Dudley Theorem 10.1.1). That probably explains part of why this is the preferred approach.
