Measure the Mutual Information between two variables that is also shared by a third variable Is there a way to measure how much of the information that two variables share is also shared by a third variable?
Say that there are three variables $X$, $Y$, $Z$ and that I need to:

*

*Predict $Y$ given $X$. Call this prediction $\hat{Y}_X$.

*Predict $Z$ given $\hat{Y}_X$. Call this prediction $\hat{Z}_{\hat{Y}_X}$.

Like this:
$X$ -> $\hat{Y}_X$ -> $\hat{Z}_{\hat{Y}_X}$
Notice that I'm not interested in estimating directly $Z$ given $X$.
Then, the question would be:
how to measure how well will I be able to predict $\hat{Z}_{\hat{Y}_X}$ without assuming any model?
Since I can't assume a model, I thought that using Mutual Information $I( , )$ might be a good idea.
Ideas that don't work:

*

*Compute $I(X, Z)$ and use it as a proxy. It doesn't work because it doesn't consider that I won't be predicting $Z$ directly with $X$.

*Compute $I(X, Y)$ and $I(Y, Z)$ and assume that if both are good, then $\hat{Z}_{\hat{Y}_X}$ will be a good prediction of $Z$. It doesn't work because it can be the case that the information that $X$ and $Y$ have in common is not the information that $Y$ and $Z$ have in common.

*Train a model that predicts $\hat{Y}_X$ and then compute $I(\hat{Y}_X, Y)$. It doesn't work because it requires assuming a model.

Maybe computing some conditional entropies might help?
Is the interaction information what I need?

I know of two points that might be relevant to your question:
First, there is the data processing inequality, which, for a Markov chain $X\to Y\to Z$ states:
$$
I(X, Y) \ge I(X, Z).
$$
This gives you an upper bound of what you can hope for when trying to predict $Z$ from $X$. Note, that this is independent of how you go about the prediction, i.e. it doesn't matter whether you try to do it directly or in two steps as how you have described it.
Second, there is a paper by Cox showing that in linear regression, it is usually better to perform the two regressions for $X\to Y$ and $Y\to Z$ and then to multiply the two regression coefficients, than to compute the regression coefficient of $X\to Z$. Here, "better" means, that the asymptotic variance of the estimated regression coefficient is smaller.
