Learning relevant features for statistical inference
We introduce an new technique to learn correlations between two types of data. The learned representation can be used to directly compute the expectations of functions over one type of data conditioned on the other, such as Bayesian estimators and their standard deviations. Specifically, our loss function teaches two neural nets to extract features representing the probability vectors of highest singular value for the stochastic map (set of conditional probabilities) implied by the joint dataset, relative to the inner product defined by the Fisher information metrics evaluated at the marginals. We test the approach using a synthetic dataset, analytical calculations, and inference on occluded MNIST images. Surprisingly, when applied to supervised learning (one dataset consists of labels), this approach automatically provides regularization and faster convergence compared to the cross-entropy objective. We also explore using this approach to discover salient independent features of a single dataset.
The paper considers how to learn correlations between two spaces, e.g., input/output, in order to generate data in one space conditioned on values from the other. This is performed by modeling features with neural networks and optimizing an objective function that maximizes a measure of correlation between the features versus learning a generative model such as a CVAE. Some illustrative examples using MNIST are provided. My decision is to reject. I think there is value in the approach, but it is hard to see clearly at the moment given that the exposition is difficult to follow and the experiments aren*t very compelling. If these issues could be addressed (concrete suggestions below), and some of the follow-on work in the last section could be performed, I think there could be a pretty interesting contribution here. *** Decision-related suggestions/questions: * Include more datasets in the experimental section. The second sentence of the introduction lists possibilities such as time series and multi-modalities that I would have been very interesting. * The first claim that there is no tunable variable in the objective function is a little hard to parse. Clearly, the rank of the low-rank approximation must be set, and the features of the two spaces need to be learned. Some clarification here would be helpful. * There are a number of unfortunate typos/grammar issues/presentation choices that really impact clarity. Some examples: * In the third paragraph under theory, *...linear spaces spanned by the probability distributions...* should probably be *...linear spaces spanned by all probability distributions...* (?) * The following sentence is a run-on. * In (8), occurrences of g_*(x_n) should be replaced with g_*(y_n). * The replacement of the (low) rank symbol, k_0, with the sample size symbol, n, in the second paragraph of section 4.1. * Introducing a *Bayesian estimator for an l^2 distance* w/o explanation. What does this mean? * How should the low-rank parameter k_0 be selected generally given that the singular value distribution may not always be useful in selecting it? * Can anything be said quantitatively or qualitatively about the sample complexity required to estimate the matrices of (8) well enough to estimate the features? * Is there an interpretation for why both spaces require the same feature dimension, k_0? *** Comments not related to decision: * It is generally good to avoid sweeping statements such as the first sentence of the introduction. Perhaps consider replacing with a simple statement on the intended goal of the paper: *...produce a useful model of correlations... for the task of data generation...* * Consider placing a concrete, motivating example prior to the theory section as it is hard to digest (from an ML perspective) without a clear context. The analytical example with the Gaussian from the supplementary material is one option. * The last statement of the paragraph under (3) needs a reference. * It seems strange to have the supervised learning experiment of 4.1 as the first experimental result of the paper since it is an unintended and unexplained side-effect of the approach. Also, the claim of *faster convergence* should be demonstrated in wall-clock time.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Learning relevant features for statistical inference
We introduce an new technique to learn correlations between two types of data. The learned representation can be used to directly compute the expectations of functions over one type of data conditioned on the other, such as Bayesian estimators and their standard deviations. Specifically, our loss function teaches two neural nets to extract features representing the probability vectors of highest singular value for the stochastic map (set of conditional probabilities) implied by the joint dataset, relative to the inner product defined by the Fisher information metrics evaluated at the marginals. We test the approach using a synthetic dataset, analytical calculations, and inference on occluded MNIST images. Surprisingly, when applied to supervised learning (one dataset consists of labels), this approach automatically provides regularization and faster convergence compared to the cross-entropy objective. We also explore using this approach to discover salient independent features of a single dataset.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



The paper considers how to learn correlations between two spaces, e.g., input/output, in order to generate data in one space conditioned on values from the other. This is performed by modeling features with neural networks and optimizing an objective function that maximizes a measure of correlation between the features versus learning a generative model such as a CVAE. Some illustrative examples using MNIST are provided. My decision is to reject. I think there is value in the approach, but it is hard to see clearly at the moment given that the exposition is difficult to follow and the experiments aren*t very compelling. If these issues could be addressed (concrete suggestions below), and some of the follow-on work in the last section could be performed, I think there could be a pretty interesting contribution here. *** Decision-related suggestions/questions: * Include more datasets in the experimental section. The second sentence of the introduction lists possibilities such as time series and multi-modalities that I would have been very interesting. * The first claim that there is no tunable variable in the objective function is a little hard to parse. Clearly, the rank of the low-rank approximation must be set, and the features of the two spaces need to be learned. Some clarification here would be helpful. * There are a number of unfortunate typos/grammar issues/presentation choices that really impact clarity. Some examples: * In the third paragraph under theory, *...linear spaces spanned by the probability distributions...* should probably be *...linear spaces spanned by all probability distributions...* (?) * The following sentence is a run-on. * In (8), occurrences of g_*(x_n) should be replaced with g_*(y_n). * The replacement of the (low) rank symbol, k_0, with the sample size symbol, n, in the second paragraph of section 4.1. * Introducing a *Bayesian estimator for an l^2 distance* w/o explanation. What does this mean? * How should the low-rank parameter k_0 be selected generally given that the singular value distribution may not always be useful in selecting it? * Can anything be said quantitatively or qualitatively about the sample complexity required to estimate the matrices of (8) well enough to estimate the features? * Is there an interpretation for why both spaces require the same feature dimension, k_0? *** Comments not related to decision: * It is generally good to avoid sweeping statements such as the first sentence of the introduction. Perhaps consider replacing with a simple statement on the intended goal of the paper: *...produce a useful model of correlations... for the task of data generation...* * Consider placing a concrete, motivating example prior to the theory section as it is hard to digest (from an ML perspective) without a clear context. The analytical example with the Gaussian from the supplementary material is one option. * The last statement of the paragraph under (3) needs a reference. * It seems strange to have the supervised learning experiment of 4.1 as the first experimental result of the paper since it is an unintended and unexplained side-effect of the approach. Also, the claim of *faster convergence* should be demonstrated in wall-clock time.
assistant
The paper considers how to learn correlations between two spaces, e.g., input/output, in order to generate data in one space conditioned on values from the other. This is performed by modeling features with neural networks and optimizing an objective function that maximizes a measure of correlation between the features versus learning a generative model such as a CVAE. Some illustrative examples using MNIST are provided. My decision is to reject. I think there is value in the approach, but it is hard to see clearly at the moment given that the exposition is difficult to follow and the experiments aren*t very compelling. If these issues could be addressed (concrete suggestions below), and some of the follow-on work in the last section could be performed, I think there could be a pretty interesting contribution here. *** Decision-related suggestions/questions: * Include more datasets in the experimental section. The second sentence of the introduction lists possibilities such as time series and multi-modalities that I would have been very interesting. * The first claim that there is no tunable variable in the objective function is a little hard to parse. Clearly, the rank of the low-rank approximation must be set, and the features of the two spaces need to be learned. Some clarification here would be helpful. * There are a number of unfortunate typos/grammar issues/presentation choices that really impact clarity. Some examples: * In the third paragraph under theory, *...linear spaces spanned by the probability distributions...* should probably be *...linear spaces spanned by all probability distributions...* (?) * The following sentence is a run-on. * In (8), occurrences of g_*(x_n) should be replaced with g_*(y_n). * The replacement of the (low) rank symbol, k_0, with the sample size symbol, n, in the second paragraph of section 4.1. * Introducing a *Bayesian estimator for an l^2 distance* w/o explanation. What does this mean? * How should the low-rank parameter k_0 be selected generally given that the singular value distribution may not always be useful in selecting it? * Can anything be said quantitatively or qualitatively about the sample complexity required to estimate the matrices of (8) well enough to estimate the features? * Is there an interpretation for why both spaces require the same feature dimension, k_0? *** Comments not related to decision: * It is generally good to avoid sweeping statements such as the first sentence of the introduction. Perhaps consider replacing with a simple statement on the intended goal of the paper: *...produce a useful model of correlations... for the task of data generation...* * Consider placing a concrete, motivating example prior to the theory section as it is hard to digest (from an ML perspective) without a clear context. The analytical example with the Gaussian from the supplementary material is one option. * The last statement of the paragraph under (3) needs a reference. * It seems strange to have the supervised learning experiment of 4.1 as the first experimental result of the paper since it is an unintended and unexplained side-effect of the approach. Also, the claim of *faster convergence* should be demonstrated in wall-clock time.