Hypothesis Testing in Unsupervised Domain Adaptation with Applications in Alzheimer's Disease
Consider samples from two different data sources {x i s } ∼ P source and {x i t } ∼ P target . We only observe their transformed versions h(x i s ) and g(x i t ), for some known function class h(·) and g(·). Our goal is to perform a statistical test checking if P source = P target while removing the distortions induced by the transformations. This problem is closely related to domain adaptation, and in our case, is motivated by the need to combine clinical and imaging based biomarkers from multiple sites and/or batches -a fairly common impediment in conducting analyses with much larger sample sizes. We address this problem using ideas from hypothesis testing on the transformed measurements, wherein the distortions need to be estimated in tandem with the testing. We derive a simple algorithm and study its convergence and consistency properties in detail, and provide lower-bound strategies based on recent work in continuous optimization. On a dataset of individuals at risk for Alzheimer's disease, our framework is competitive with alternative procedures that are twice as expensive and in some cases operationally infeasible to implement.
This paper introduces a new method for domain adaptation that relies the Maximum Mean Discrepancy (MMD). Transformations of both the source and target distributions are learned while jointly minimizing the MMD between the transformed versions. This yields a statistical test of whether the predefined function class is rich enough to have transformations that make the two distributions indistinguishable, and this test is shown to be consistent.  The method is illustrated on batches of data from an Alzheimer's disease study, showing that shifting the distribution achieves some improvements in analytical performance.  
The paper presents an interesting and smart way of performing covariate shift by aiming to make the two distributions indistinguishable by minimizing MMD. The paper however could benefit of more clarity and completeness so it can make impact.   In terms of the applicability of this approach, the authors talk about the importance of being able to perform statistical tests and not just optimize the performance of a classifier. The bounds that they derive for their statistical test is useful to know how big the sample size should be to perform an appropriate shift. However, this doesn't say much on how it affects the scientific questions asked in the experiment, which need different statistical test. From the two sites where the data originates, we might (for example) have two types of patients. How does this shift help us with identifying them? Table 1 shows the answer to a similar question where the hippocampal volume is better predicted by the morphed CSF data, but more tests such as this one are required to believe that the method is useful. What if the question of interest was to find the CSF proteins that are most predictive, how would the shifting of the covariates affect our ability to answer the question?  More experiments, as well as a better explanation of the problem would make the paper stronger. For example, what if the data from the two sites were unevenly distributed such as they had different proportions of the possible phenotypes. The variance we would see would not be a consequence of the measurement only, but also reflects population variance. The proposed measure could however homogenize the two batches and introduce some error. Perhaps these questions are hard to answer theoretically but more experimental results could help.  It would be interesting to also mention how this method could be applied to more than two batches.