Multivariate Regression with Calibration ⇤
We propose a new method named calibrated multivariate regression (CMR) for fitting high dimensional multivariate regression models. Compared to existing methods, CMR calibrates the regularization for each regression task with respect to its noise level so that it is simultaneously tuning insensitive and achieves an improved finite-sample performance. Computationally, we develop an efficient smoothed proximal gradient algorithm which has a worst-case iteration complexity O(1/✏), where ✏ is a pre-specified numerical accuracy. Theoretically, we prove that CMR achieves the optimal rate of convergence in parameter estimation. We illustrate the usefulness of CMR by thorough numerical simulations and show that CMR consistently outperforms other high dimensional multivariate regression methods. We also apply CMR on a brain activity prediction problem and find that CMR is as competitive as the handcrafted model created by human experts.
This paper proposed a new method called calibrated multivariate regression (CMR) for fitting high dimensional multivariate regression models. The assumption is that the noise matrix has an uncorrelated structure, and different regression tasks have different noise variances. Instead of using the same tuning parameter lambda for all the regression tasks, CMR calibrates different tasks by solving a penalized weighted least square problem weights define in (2.3) (i.e., some kind of estimate of noise standard deviation). The paper is well-structured and clearly-written. The appendix contains a lot of technical details. The idea of CMR formulation is technically sound. The proposed computational algorithms and demonstrated statistical properties make sense at a high level, although I did not check every step of the derivation and proof given the limited expertise in this field. The empirical study on simulated and real data yielded promising results.Line 129: Should "weighted least square program" read "weighted least square problem"?Another straightforward approach to calibrate the parameter lambda is to run multiple regression on each response variable separately. How does this compare with the proposed CMR? It would be interesting to compare them on both simulated and real data. Sometimes simple models might yield better or more interpretable results on real data than more advanced ones.