Variational lower bounds on mutual information based on nonextensive statistical mechanics
This paper aims to address the limitations of mutual information estimators based on variational optimization. By redefining the cost using generalized functions from nonextensive statistical mechanics we raise the upper bound of previous estimators and enable the control of the bias variance trade off. Variational based estimators outperform previous methods especially in high dependence high dimensional scenarios found in machine learning setups. Despite their performance, these estimators either exhibit a high variance or are upper bounded by log(batch size). Our approach inspired by nonextensive statistical mechanics uses different generalizations for the logarithm and the exponential in the partition function. This enables the estimator to capture changes in mutual information over a wider range of dimensions and correlations of the input variables whereas previous estimators saturate them.
This paper tries to improve the limitations of previous variational bounds for mutual information estimation, where it uses generalized functions from nonextensive statistical mechanics to get generalized better bounds for the logarithm and the exponential in the partition function. Experiments on simple synthetic data are performed to show the effectiveness, but these experiments are too simple, more thorough experiments on real data sets should be conducted. There are some confusions in the writing. In Equation (4), the last term is a Monte Carlo sample average by K samples to approximate the partition function Z, so approximate symbol should be used for last term instead of = symbol. Similarly in Equation (5), the last inequality is not really an inequality since this is a Monte Carlo sample average by K samples to approximate the terms in the previous inequality. This is also true for equation or inequality of (6), (18)-(20). In fact this is a problem that appears in Poor and Oord*s papers. On page 2, second line K is the batch size, then the first line after equation (4), K is the number of samples. Please make it consistent. On page 3, the first and second paragraph, it is stated *I_{NWJ} estimator performs well in high dimensional high dependence scenarios but exhibits high variance due to the exponential function in the partition function which amplifies the critic errors. One way to reduce the variance is to make the critic depend on multiple samples and use a Monte Carlo estimator for the partition function.* Please explain why I_{NWJ} estimator exhibits high variance, is it observed by experiments, or it*s been proved mathematically? Similarly why making the critic depend on multiple samples and use a Monte Carlo estimator for the partition function can reduce the variance. I see these kind of statements often appear on papers, I know Poore*s paper provide experimental evidence of the validness, but I don*t see any mathematical proof. The way to cite the references is not right, for example, McAllester & Statos (2018), the correct one is (McAllester & Statos 2018).
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Variational lower bounds on mutual information based on nonextensive statistical mechanics
This paper aims to address the limitations of mutual information estimators based on variational optimization. By redefining the cost using generalized functions from nonextensive statistical mechanics we raise the upper bound of previous estimators and enable the control of the bias variance trade off. Variational based estimators outperform previous methods especially in high dependence high dimensional scenarios found in machine learning setups. Despite their performance, these estimators either exhibit a high variance or are upper bounded by log(batch size). Our approach inspired by nonextensive statistical mechanics uses different generalizations for the logarithm and the exponential in the partition function. This enables the estimator to capture changes in mutual information over a wider range of dimensions and correlations of the input variables whereas previous estimators saturate them.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



This paper tries to improve the limitations of previous variational bounds for mutual information estimation, where it uses generalized functions from nonextensive statistical mechanics to get generalized better bounds for the logarithm and the exponential in the partition function. Experiments on simple synthetic data are performed to show the effectiveness, but these experiments are too simple, more thorough experiments on real data sets should be conducted. There are some confusions in the writing. In Equation (4), the last term is a Monte Carlo sample average by K samples to approximate the partition function Z, so approximate symbol should be used for last term instead of = symbol. Similarly in Equation (5), the last inequality is not really an inequality since this is a Monte Carlo sample average by K samples to approximate the terms in the previous inequality. This is also true for equation or inequality of (6), (18)-(20). In fact this is a problem that appears in Poor and Oord*s papers. On page 2, second line K is the batch size, then the first line after equation (4), K is the number of samples. Please make it consistent. On page 3, the first and second paragraph, it is stated *I_{NWJ} estimator performs well in high dimensional high dependence scenarios but exhibits high variance due to the exponential function in the partition function which amplifies the critic errors. One way to reduce the variance is to make the critic depend on multiple samples and use a Monte Carlo estimator for the partition function.* Please explain why I_{NWJ} estimator exhibits high variance, is it observed by experiments, or it*s been proved mathematically? Similarly why making the critic depend on multiple samples and use a Monte Carlo estimator for the partition function can reduce the variance. I see these kind of statements often appear on papers, I know Poore*s paper provide experimental evidence of the validness, but I don*t see any mathematical proof. The way to cite the references is not right, for example, McAllester & Statos (2018), the correct one is (McAllester & Statos 2018).
assistant
This paper tries to improve the limitations of previous variational bounds for mutual information estimation, where it uses generalized functions from nonextensive statistical mechanics to get generalized better bounds for the logarithm and the exponential in the partition function. Experiments on simple synthetic data are performed to show the effectiveness, but these experiments are too simple, more thorough experiments on real data sets should be conducted. There are some confusions in the writing. In Equation (4), the last term is a Monte Carlo sample average by K samples to approximate the partition function Z, so approximate symbol should be used for last term instead of = symbol. Similarly in Equation (5), the last inequality is not really an inequality since this is a Monte Carlo sample average by K samples to approximate the terms in the previous inequality. This is also true for equation or inequality of (6), (18)-(20). In fact this is a problem that appears in Poor and Oord*s papers. On page 2, second line K is the batch size, then the first line after equation (4), K is the number of samples. Please make it consistent. On page 3, the first and second paragraph, it is stated *I_{NWJ} estimator performs well in high dimensional high dependence scenarios but exhibits high variance due to the exponential function in the partition function which amplifies the critic errors. One way to reduce the variance is to make the critic depend on multiple samples and use a Monte Carlo estimator for the partition function. * Please explain why I_{NWJ} estimator exhibits high variance, is it observed by experiments, or it*s been proved mathematically? Similarly why making the critic depend on multiple samples and use a Monte Carlo estimator for the partition function can reduce the variance. I see these kind of statements often appear on papers, I know Poore*s paper provide experimental evidence of the validness, but I don*t see any mathematical proof. The way to cite the references is not right, for example, McAllester & Statos (2018), the correct one is (McAllester & Statos 2018).