It is all in the noise: Efficient multi-task Gaussian process inference with structured residuals
Multi-task prediction methods are widely used to couple regressors or classification models by sharing information across related tasks. We propose a multi-task Gaussian process approach for modeling both the relatedness between regressors and the task correlations in the residuals, in order to more accurately identify true sharing between regressors. The resulting Gaussian model has a covariance term in form of a sum of Kronecker products, for which efficient parameter inference and out of sample prediction are feasible. On both synthetic examples and applications to phenotype prediction in genetics, we find substantial benefits of modeling structured noise compared to established alternatives.
Update after reading the authors' rebuttal: 
      Very good paper but I strongly recommend the authors to consider 
      all the suggestions below that will improve the readability of the paper. 
      ===== 1. Summary: This paper presents a multi-task 
      Gaussian process regression approach where the covariance of the main 
      process (signal) decomposes as a product of a covariance between tasks and 
      a covariance between inputs (sample covariance). It is assumed that all 
      the training outputs are observed at all inputs, which leads to a 
      Kronecker product covariance. Noisy observations are modeled via a 
      structured process and this is the main contribution of the paper. While 
      previous work on multi-task GP approaches with Kronecker covariances has 
      considered iid noise in order to carry out efficient computations, this 
      paper shows that it is possible to consider a noise process with Kronecker 
      structure, while maintaining efficient computations. In other words, as in 
      the iid noise case, one never has to compute a Kronecker product and hence 
      computations are O(N^3 + T^3) instead of O(N^3T^3). This is achieved by 
      whitening the noise process and projecting the (noiseless) covariance of 
      the system into the eigen-basis of the noise covariance (scaled by the 
      eigenvalues). Their experiments show that the proposed 
      structured-noise multi-task GP approach outperforms the baseline iid-noise 
      multi-task GP method and independent GPs on synthetic data and real 
      applications. 2. Quality The paper seems 
      technically sound once one realizes how to fix the notational 
      inconsistencies between section 2 and section 3 (please see item 4 below 
      regarding Clarity). The main claim that one can have structured noise in 
      multi-task GP models with product covariances while still maintaining 
      computational efficiency (compared to the naive approach) is well 
      supported with the mathematical derivations in section 3 and with the 
      experimental results in section 4. With respect to the results, 
      although it is obvious that the naive approach would be much worse in 
      terms of computation compared to the efficient approach, it is still 
      helpful to see the comparison in Figure 1 so one can take into 
      consideration the possible overhead in implementing the GP-KS method. 
      However, there are a few deficiencies that need to be pointed out: 
      (a) There is a rank-1 parameterization of C and \Sigma. However, 
      it is unclear how the parameter \sigma in line 246 was set. This parameter 
      is important as it allows that the algorithm can actually run but also 
      that one does not over-smooth the process leading to poor generalization 
      performance. (b) It is completely unclear what \Omega is actually 
      used (please see Clarity below). This should be explicitly said in the 
      experiments. From section 2, it can be inferred that \Omega = I_{NN}. But 
      the reader should not be guessing about something that must be explicit. 
      (c) Line 328: "Large-scale prediction .." : 123 tasks and 218 
      samples are very far from what can be considered as large-scale. 
      (d) In the experiments regarding the prediction of gene expression 
      in yeast, it looks like the preprocessing and filtering of the dataset 
      does favor the proposed method (which may have problems with 
      identifiability) as genes with low signal and low noise are discarded. The 
      authors should provide comments on this. (e) The authors have not 
      analyzed possible weaknesses of their method. In particular, interpreting 
      the results in Figure 3 is a bit misleading as it seems that their method 
      has high levels of unidentifiability. Why is it possible to interpret the 
      results? There may be completely different qualitative solutions that lead 
      to similar quantitative performance. Is identifiability an issue during 
      optimization? (f) Multi-task settings usually compare to a single 
      GP for all tasks (i.e. pool GP). This baseline is missing. 3. 
      Clarity In terms of language use, the paper is relatively well 
      written. However, there are quite a few notational inconsistencies that 
      may push this paper below the threshold. For example: (a) The same 
      symbol (k) in Equation 1 is used for both covariance (sample and task) 
      functions and this is completely inconsistent with the following notation 
      in the paper (C, R). On the same equation, is this really the covariance 
      between the noisy outputs y or should it be between their corresponding 
      noiseless latent functions? (b) Sometimes C_{TT}, R_{NN}, etc are 
      used and other times C, T, are used (c) This is inconsistency is 
      crucial to understand the paper: I_{NN} is used in Equation 3 but then 
      \Omega is introduced in section 3 without explaining what it refers 
      to. Is \Omega = I_{NN}? (d) K is undefined before being used in 
      Equation 6 (e) Equation 9 does not make sense. It goes from a 
      vector on the line above to a matrix. Is there a Vec operator missing? 
      4. Originality The approach to multi-task GP 
      regression differs from most previous work in the way the noise process is 
      modeled (structured noise compared to idd noise) while maintaining 
      efficiency during inference. These types of processes have been considered 
      before for example by Zhang (2007) but efficient computations were not 
      explicitly done when considering the specific case of two processes. 
      Without committing to a specific approach for flexible multi-task 
      models, it seems necessary to at least mention how this works compare to 
      the Gaussian process regression network framework in [6]. 
      Additionally, sec 2.2 is obvious and should be omitted. The case C 
      = \Sigma leads to C \otimes (R+I), which is the same proof shown in 
      previous work [1]. 5. Significance The contribution of 
      this paper is relatively significant in that it shows that it is possible 
      to do efficient computation in these types of models when a sum of two 
      Kronecker products is present. This can be exploited in scenarios 
      different to the regression setting. However, in terms of the original 
      motivation, i.e. multi-task regression, there are other more flexible 
      models [6] for which inference is still better than the naive approach 
      (N^3 T^3). 6. Additional comments (a) Abstract, lines 
      18-20: This is not true. References: Hao Zhang. 
      Maximum-likelihood estimation for multivariate spatial linear 
      coregionalization models. Environmetrics, 18(2):125–139, 2007