Given two views of data, we consider the problem of finding the features of one view which can be most faithfully inferred from the other. We find that these are also the most correlated variables in the sense of deep canonical correlation analysis (DCCA). Moreover, we show that these variables can be used to construct a non-parametric representation of the implied joint probability distribution. This representation can be used to compute the expectations of functions over one view of data conditioned on the other, such as Bayesian estimators and their standard deviations. We test the approach using inference on occluded MNIST images, and show that our representation contains multiple modes. Surprisingly, when applied to supervised learning (one dataset consists of labels), this approach automatically provides regularization and faster convergence compared to the cross-entropy objective. We also explore using this approach to discover salient independent variables of a single dataset.
The paper considers how to learn correlations between two spaces, e.g., input/output, in order to generate data in one space conditioned on values from the other. This is performed by modeling features with neural networks and optimizing an objective function that maximizes a measure of correlation between the features versus learning a generative model such as a CVAE. Some illustrative examples using MNIST are provided.
My decision is to reject. I think there is value in the approach, but it is hard to see clearly at the moment given that the exposition is difficult to follow and the experiments aren't very compelling. If these issues could be addressed (concrete suggestions below), and some of the follow-on work in the last section could be performed, I think there could be a pretty interesting contribution here.
***
Decision-related suggestions/questions:
* Include more datasets in the experimental section. The second sentence of the introduction lists possibilities such as time series and multi-modalities that I would have been very interesting.
* The first claim that there is no tunable variable in the objective function is a little hard to parse. Clearly, the rank of the low-rank approximation must be set, and the features of the two spaces need to be learned. Some clarification here would be helpful. 
* There are a number of unfortunate typos/grammar issues/presentation choices that really impact clarity. Some examples:
	* In the third paragraph under theory, "...linear spaces spanned by the probability distributions..." should probably be "...linear spaces spanned by all probability distributions..." (?)
	* The following sentence is a run-on. 
	* In (8), occurrences of g_*(x_n) should be replaced with g_*(y_n).
	* The replacement of the (low) rank symbol, k_0, with the sample size symbol, n, in the second paragraph of section 4.1.
	* Introducing a "Bayesian estimator for an l^2 distance" w/o explanation. What does this mean?
* How should the low-rank parameter k_0 be selected generally given that the singular value distribution may not always be useful in selecting it?
* Can anything be said quantitatively or qualitatively about the sample complexity required to estimate the matrices of (8) well enough to estimate the features?
* Is there an interpretation for why both spaces require the same feature dimension, k_0?
***
Comments not related to decision:
* It is generally good to avoid sweeping statements such as the first sentence of the introduction. Perhaps consider replacing with a simple statement on the intended goal of the paper: "...produce a useful model of correlations... for the task of data generation..."
* Consider placing a concrete, motivating example prior to the theory section as it is hard to digest (from an ML perspective) without a clear context. The analytical example with the Gaussian from the supplementary material is one option.
* The last statement of the paragraph under (3) needs a reference.
* It seems strange to have the supervised learning experiment of 4.1 as the first experimental result of the paper since it is an unintended and unexplained side-effect of the approach. Also, the claim of "faster convergence" should be demonstrated in wall-clock time.