Hypothesis Transfer Learning via Transformation Functions
We consider the Hypothesis Transfer Learning (HTL) problem where one incorporates a hypothesis trained on the source domain into the learning procedure of the target domain. Existing theoretical analysis either only studies specific algorithms or only presents upper bounds on the generalization error but not on the excess risk. In this paper, we propose a unified algorithm-dependent framework for HTL through a novel notion of transformation function, which characterizes the relation between the source and the target domains. We conduct a general risk analysis of this framework and in particular, we show for the first time, if two domains are related, HTL enjoys faster convergence rates of excess risks for Kernel Smoothing and Kernel Ridge Regression than those of the classical non-transfer learning settings. Experiments on real world data demonstrate the effectiveness of our framework.
The paper presents a supervised non-parametric hypothesis transfer learning (HTL) approach for regression and its analysis, aimed at the cases where one has plenty of training data coming from the source task and few examples from the target one. The paper makes an assumption that the source and the target regression functions are related through so called transformation function (TF). The TF is assumed to have some parametric form (e.g. linking regressions functions linearly) and the goal of an algorithm is to recover its parameters. Once these parameters are learned, the hypothesis trained on the source task can be transformed to the hypothesis designated for the target task. The paper proposes two ways for estimation of these parameters, that is through kernel smoothing and kernel ridge regression. For both cases the paper presents consistency bounds, showing that one can have an improvement in the exponent of the non-parametric rate once the TF is smoother, in Holder sense, than the target regression function. In addition, the paper argues that one can run model selection over the class of TF through cross-validation and shows some bounds to justify this, which is not particularly surprising. Finally, the paper concludes with numerical experiments on two regression tasks on the real-world small-scale datasets.
The rates stated in main theorems, 2 and 3 appear to make sense. One can see this from theorem 1, which is used to prove both, and essentially states an excess risk as a decomposition into the sum of excess risk bounds for source/TF regression functions. The later are bounded through well established excess risk bounds from the literature.
I do not think this is a breakthrough paper, but it has a novel point as it addresses the theory of HTL in the non-parametric setting because it studies the Bayes risk rates rather than generalization bounds as was done previously. 
The paper has a novel bit as it studies non-parametric consistency rates of HTL, while previous theoretical works focused on the risk bounds. As one should expect, the improvement then occurs in the right place, that is in the exponent of the rate, rather than in some additive or multiplicative term. At the same time the algorithm appears to be simple and easy to implement. This makes this paper a good contribution to the literature on HTL. Also, the paper is quite well positioned in the related work. Unfortunately, it doesn't compare to previous HTL approaches experimentally and experimental evaluation is, frankly, quite modest, despite the simplicity of the algorithm.