Variational approaches based on neural networks are showing promise for estimating mutual information (MI) between high dimensional variables. However, they can be difficult to use in practice due to poorly understood bias/variance tradeoffs. We theoretically show that, under some conditions, estimators such as MINE exhibit variance that could grow exponentially with the true amount of underlying MI. We also empirically demonstrate that existing estimators fail to satisfy basic self-consistency properties of MI, such as data processing and additivity under independence. Based on a unified perspective of variational approaches, we develop a new estimator that focuses on variance reduction. Empirical results on standard benchmark tasks demonstrate that our proposed estimator exhibits improved biasvariance trade-offs on standard benchmark tasks.
Summary:
As the title suggests the paper focusses mostly on a negative result:
Mutual information (MI) estimators obtained by variational methods have severe
limitations that make them potentially not useful for down stream tasks.
Besides highlighting the problems with variational MI estimators the authors
suggest a modification to slightly improve the performance of MI estimators
based on partition functions by reducing their variance when MI is high.
The authors give a good overview / introduction of various approaches to
variational MI estimation by discriminative and generative methods.
Generally, MI estimation involves the estimation of the KL divergence between
the joint distribution and the product of the marginals.
The authors present a unifying view on the different approaches that optimizes
the log density ratio required for the KL divergence over the space of
log density ratios.
Discriminative approaches model the density ratio directly (through e.g. neural
network models) and generative approaches model the separate densities
(as generative models where it is possible to evaluate the (conditional)
probabilities / likelihoods of the data generating process).
The authors prove that discriminative approaches that are based on the partition
function approach suffer from high variance where mutual information is high
(Theorem 2).
The estimator based on a finite sample has high variance even if the density
ratio approximation is correct.
(The partition function approach is a way of staying constrained to the log
density ratio function space.)
This high variance problem is something that has previously been observed
empirically and is the main theoretical point that is being made about
limitations of MI estimators.
In order to slightly alleviate the problem of high variance the authors suggest
a way of biasing MI estimators by clipping the density ratio estimates
through a constant chosen as a hyper-parameter.
They prove that their clipping approach reduces variance and therefore
introduces a bias variance tradeoff.
In their later experiments the clipped version of the discriminative approach
performs much better in terms of variance than without clipping and also better
than a generative approach.
In order to empirically evaluate the quality of MI estimators the authors
suggest three criteria that they call self-consistency:
(i) independence, (ii) data processing, (iii) additivity
Self-consistency is evaluated experimentally on images where mutual information
is computed between original image and image with part covered.
The authors claim and experimentally show that discriminative approaches fail
in (iii) and generative approaches fail in (i), (ii).
Overall, variational MI approaches do not satisfy self-consistency.
Evaluation:
I suggest to accept the paper.
The theoretical contribution of showing the variance limitation of
discriminative approaches seems significant.
That insight leads to the idea that clipping can be a useful bias that
significantly reduces variance without making the already biased anyways
results much worst in the experiments.
However, I also feel like the paper is not yet as focused as it could be.
It contains many concepts that could need a little bit more space.
Suggestions:
- Page 2: Nitpick, but in the definition of $L^p(Q)$ using $\colon$ twice is
	not super readable on the first read
- Page 2: In the definition of $I_{BA}$ clearify whether $p(x)$ is a marginal
	or a joint density (as $P$ is the cumulative joint)
- Page 3: In Theorem 1 what is the definition of $P \ll Q$?
	This suggests an order on the space of measures $\mathcal P$?
- Page 3: "Obtain an density ratio estimate" -> Obtain a density ratio estimate