Understanding the Limitations of Variational Mutual Information Estimators
Variational approaches based on neural networks are showing promise for estimating mutual information (MI) between high dimensional variables. However, they can be difficult to use in practice due to poorly understood bias/variance tradeoffs. We theoretically show that, under some conditions, estimators such as MINE exhibit variance that could grow exponentially with the true amount of underlying MI. We also empirically demonstrate that existing estimators fail to satisfy basic self-consistency properties of MI, such as data processing and additivity under independence. Based on a unified perspective of variational approaches, we develop a new estimator that focuses on variance reduction. Empirical results on standard benchmark tasks demonstrate that our proposed estimator exhibits improved bias-variance trade-offs on standard benchmark tasks.
This paper relates most existing variational mutual information estimators to density ratio estimation and uses this to show high variance of certain categories of estimators; this motivates a new SMILE estimator that uses clipping to explicitly control the variance of the estimator (while potentially increasing bias). The paper also conducts some experiments on how well mutual information estimators satisfy some of the core properties of mutual information in practice. The core result of Theorem 2, although is proof is very simple once the problem has been phrased appropriately, is enlightening. The self-consistency tests are also sensible and important, although it would be nice to understand better which of these properties are truly important for various uses of mutual information in the literature. The proposed SMILE estimator is natural. But: - As , the variance bound of Theorem 4 naturally becomes infinite (as we*d expect). But we*d hope that the bias would become 0, when in fact Theorem 3 only shows in this case that the bias is at most S. Of course it should be true that ; can we fix Theorem 3 so that it recovers this fact? Then the biggest question not really addressed by this paper is: how do you set ? - If we know the (approximate) value of , then we can presumably trade off between the bias and variance terms of Theorem 3 and 4 to find the most effective value for . (Indeed, it might be nice to write a corollary showing that if you set to something depending on and some upper bound on , you achieve some rate in L2 risk.) - Is it possible to relate the value of to, say, the suboptimality in cross-entropy loss of the classifier used to obtain (when getting that way)? - Is it possible to get a probabilistic upper bound on , either with some other technique or perhaps a simple Hoeffding/Bernstein-type bound on the SMILE estimator with some initial value of , and then use that to choose the rate for ? - Is there some other practical scheme (even if it doesn*t admit bounds) to set to control the bias, given that it depends on ? Smaller points and minor suggestions: - In the paragraph following Theorem 2, it is first of all strange to say that *alternative procedures to obtaining the density ratio estimates will not alleviate the high variance issue* when indeed your proposal does just that. The last sentence of this paragraph clarifies the issue, but still makes sound *bad,* when in fact it*s a trade-off; this could be rephrased more clearly. - In Theorem 3: I think writing would be clearer than as you have it. - In the multivariate Gaussian experiments, Section 6.1: why do you effectively warm-start the estimator for higher mutual informations? Is it just so you can put things in one plot per method? I think it would be more natural to train each different problem from scratch; you can still use the same plot format if you like (with additional vertical lines between problems or something), but it*s not clear whether the behavior of the estimator is affected by this choice. - Why are the flow-based results not shown in Figure 2 / Table 2? - In Section 6.2 when setting up the three evaluation settings, it would be good to have a forward reference to Figure 3 (since these settings are perhaps easier to understand visually).
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Understanding the Limitations of Variational Mutual Information Estimators
Variational approaches based on neural networks are showing promise for estimating mutual information (MI) between high dimensional variables. However, they can be difficult to use in practice due to poorly understood bias/variance tradeoffs. We theoretically show that, under some conditions, estimators such as MINE exhibit variance that could grow exponentially with the true amount of underlying MI. We also empirically demonstrate that existing estimators fail to satisfy basic self-consistency properties of MI, such as data processing and additivity under independence. Based on a unified perspective of variational approaches, we develop a new estimator that focuses on variance reduction. Empirical results on standard benchmark tasks demonstrate that our proposed estimator exhibits improved bias-variance trade-offs on standard benchmark tasks.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



This paper relates most existing variational mutual information estimators to density ratio estimation and uses this to show high variance of certain categories of estimators; this motivates a new SMILE estimator that uses clipping to explicitly control the variance of the estimator (while potentially increasing bias). The paper also conducts some experiments on how well mutual information estimators satisfy some of the core properties of mutual information in practice. The core result of Theorem 2, although is proof is very simple once the problem has been phrased appropriately, is enlightening. The self-consistency tests are also sensible and important, although it would be nice to understand better which of these properties are truly important for various uses of mutual information in the literature. The proposed SMILE estimator is natural. But: - As , the variance bound of Theorem 4 naturally becomes infinite (as we*d expect). But we*d hope that the bias would become 0, when in fact Theorem 3 only shows in this case that the bias is at most S. Of course it should be true that ; can we fix Theorem 3 so that it recovers this fact? Then the biggest question not really addressed by this paper is: how do you set ? - If we know the (approximate) value of , then we can presumably trade off between the bias and variance terms of Theorem 3 and 4 to find the most effective value for . (Indeed, it might be nice to write a corollary showing that if you set to something depending on and some upper bound on , you achieve some rate in L2 risk.) - Is it possible to relate the value of to, say, the suboptimality in cross-entropy loss of the classifier used to obtain (when getting that way)? - Is it possible to get a probabilistic upper bound on , either with some other technique or perhaps a simple Hoeffding/Bernstein-type bound on the SMILE estimator with some initial value of , and then use that to choose the rate for ? - Is there some other practical scheme (even if it doesn*t admit bounds) to set to control the bias, given that it depends on ? Smaller points and minor suggestions: - In the paragraph following Theorem 2, it is first of all strange to say that *alternative procedures to obtaining the density ratio estimates will not alleviate the high variance issue* when indeed your proposal does just that. The last sentence of this paragraph clarifies the issue, but still makes sound *bad,* when in fact it*s a trade-off; this could be rephrased more clearly. - In Theorem 3: I think writing would be clearer than as you have it. - In the multivariate Gaussian experiments, Section 6.1: why do you effectively warm-start the estimator for higher mutual informations? Is it just so you can put things in one plot per method? I think it would be more natural to train each different problem from scratch; you can still use the same plot format if you like (with additional vertical lines between problems or something), but it*s not clear whether the behavior of the estimator is affected by this choice. - Why are the flow-based results not shown in Figure 2 / Table 2? - In Section 6.2 when setting up the three evaluation settings, it would be good to have a forward reference to Figure 3 (since these settings are perhaps easier to understand visually).
assistant
This paper relates most existing variational mutual information estimators to density ratio estimation and uses this to show high variance of certain categories of estimators; this motivates a new SMILE estimator that uses clipping to explicitly control the variance of the estimator (while potentially increasing bias). The paper also conducts some experiments on how well mutual information estimators satisfy some of the core properties of mutual information in practice. The core result of Theorem 2, although is proof is very simple once the problem has been phrased appropriately, is enlightening. The self-consistency tests are also sensible and important, although it would be nice to understand better which of these properties are truly important for various uses of mutual information in the literature. The proposed SMILE estimator is natural. But: - As , the variance bound of Theorem 4 naturally becomes infinite (as we*d expect). But we*d hope that the bias would become 0, when in fact Theorem 3 only shows in this case that the bias is at most S. Of course it should be true that ; can we fix Theorem 3 so that it recovers this fact? Then the biggest question not really addressed by this paper is: how do you set ? - If we know the (approximate) value of , then we can presumably trade off between the bias and variance terms of Theorem 3 and 4 to find the most effective value for . (Indeed, it might be nice to write a corollary showing that if you set to something depending on and some upper bound on , you achieve some rate in L2 risk.) - Is it possible to relate the value of to, say, the suboptimality in cross-entropy loss of the classifier used to obtain (when getting that way)? - Is it possible to get a probabilistic upper bound on , either with some other technique or perhaps a simple Hoeffding/Bernstein-type bound on the SMILE estimator with some initial value of , and then use that to choose the rate for ? - Is there some other practical scheme (even if it doesn*t admit bounds) to set to control the bias, given that it depends on ? Smaller points and minor suggestions: - In the paragraph following Theorem 2, it is first of all strange to say that *alternative procedures to obtaining the density ratio estimates will not alleviate the high variance issue* when indeed your proposal does just that. The last sentence of this paragraph clarifies the issue, but still makes sound *bad,* when in fact it*s a trade-off; this could be rephrased more clearly. - In Theorem 3: I think writing would be clearer than as you have it. - In the multivariate Gaussian experiments, Section 6.1: why do you effectively warm-start the estimator for higher mutual informations? Is it just so you can put things in one plot per method? I think it would be more natural to train each different problem from scratch; you can still use the same plot format if you like (with additional vertical lines between problems or something), but it*s not clear whether the behavior of the estimator is affected by this choice. - Why are the flow-based results not shown in Figure 2 / Table 2? - In Section 6.2 when setting up the three evaluation settings, it would be good to have a forward reference to Figure 3 (since these settings are perhaps easier to understand visually).