Canonical Correlation Analysis (CCA) is a ubiquitous technique that shows promising performance in multi-view learning problems. Due to the conjugacy of the prior and the likelihood, probabilistic CCA (PCCA) presents the posterior with an analytic solution, which provides probabilistic interpretation for classic linear CCA. As the multi-view data are usually complex in practice, nonlinear mappings are adopted to capture nonlinear dependency among the views. However, the interpretation provided in PCCA cannot be generalized to this nonlinear setting, as the distribution assumptions on the prior and the likelihood makes it restrictive to capture nonlinear dependency. To overcome this bottleneck, in this paper, we provide a novel perspective for CCA based on implicit distributions. Specifically, we present minimum Conditional Mutual Information (CMI) as a new criteria to capture nonlinear dependency for multi-view learning problem. To eliminate the explicit distribution requirement in direct estimation of CMI, we derive an objective whose minimization implicitly leads to the proposed criteria. Based on this objective, we present an implicit probabilistic formulation for CCA, named Implicit CCA (ICCA), which provides a flexible framework to design CCA extensions with implicit distributions. As an instantiation, we present adversarial CCA (ACCA), a nonlinear CCA variant which benefits from consistent encoding achieved by adversarial learning. Quantitative correlation analysis and superior performance on cross-view generation task demonstrate the superiority of the proposed ACCA.
I don't quite see how the proposed approach addresses non-linear canonical correlation analysis. In particular:
1) The main motivation is the minimization of the conditional mutual information I(X;Y|Z), where X and Y correspond to the two views and Z is latent. First of all, what uncertainty does this expression has when X and Y are observations and Z is given? My understanding is that the main objective of any CCA problem should be to find some transformations, say f(X) and g(Y), with some (to be defined) desirable properties. For example, these would correspond to linear transformations, say Ax and By, for classical CCA. Therefore, should not one be interested in minimizing something like I(f(X);g(Y)|Z)?
2) Assuming that the minimization of the conditional mutual information I(X;Y|Z) would be the goal, I don't quite see why the formulation in equation (6) would actually be equivalent (or be some reasonable approximation)? 
3) It is well known that differential entropy can be negative (e.g., Cover and Thomas, 2006). Why would the conditional mutual information in equation (4) be non-negative? Alternatively, what would negative values of I(X;Y|Z) mean in the CCA context? My understanding is that one should be interested in minimizing I(X;Y|Z), or its variants with transformations, in *absolute value* to ensure some closeness to conditional independence.
4) Expressions in equation (5)-(6) are general and hold with no assumptions whatsoever for any random variables X, Y, Z (given the expectations/integrals exist). It is therefore not clear what are the variables of this minimization problem? (parameters? but what is the parametric model?)
5) Assuming solving (6) is the goal, this formulation as mentioned by the authors is actually is quite a challenging problem involving latent variables. Some form of this approach explanation would 
I can not quite see how the proposed adversarial version would correct or supplement any of these questions.
Other comments:
1) It would be appropriate to cite the probabilistic CCA paper by Bach and Jordan (2005); a better citation for classical CCA would be Hotelling (1936).
2) I find the multiple mentioning of the *non-linear* (in-)dependence confusing. Is this in statistical sense? And how exactly is this related to CCA? Does it have anything to do with the fact that the third and higher order cumulants are zero only for independent variables unless they are Gaussian? Moreover, does this linear independence have any connection with the non-linearity of the proposed CCA approach?
3) What exactly is the *linear correlation criterion* and how does it enter the classical CCA or PCCA formulation (Introduction; bullet point 2)?
4) It would be helpful to introduce the original CCA problem emphasizing that each view, X and Y, are *different* linear transformation of *the same* latent codes z. Moreover, the full description of the models (classical CCA/ PCCA) wouldn't take more than one-two paragraphs and would help the readers to avoid any misunderstanding.
5) Are any assumptions necessary to ensure existence?