Efficient Modeling of Latent Information in Supervised Learning using Gaussian Processes
Often in machine learning, data are collected as a combination of multiple conditions, e.g., the voice recordings of multiple persons, each labeled with an ID. How could we build a model that captures the latent information related to these conditions and generalize to a new one with few data? We present a new model called Latent Variable Multiple Output Gaussian Processes (LVMOGP) that allows to jointly model multiple conditions for regression and generalize to a new condition with a few data points at test time. LVMOGP infers the posteriors of Gaussian processes together with a latent space representing the information about different conditions. We derive an efficient variational inference method for LVMOGP for which the computational complexity is as low as sparse Gaussian processes. We show that LVMOGP significantly outperforms related Gaussian process methods on various tasks with both synthetic and real data.
# Update after author feedback and reviewer discussion
I thank the authors for feedback. I maintain my assessment and do not recommend publication at this stage.
The core contribution is representing conditions with latent variables, and deriving a VI algorithm to cope with intractibility. This is interesting, but the discussion around it could be much improved. 
Some possible improvements are addressed in the author feedback, eg I'm not sure how Fig 1 could have been understood without the complementary explanation brought up in the feedback. 
Beyond what has been addressed in the author feedback, some work is needed to make this paper appealing (which the idea under study, the method and the results seem to call for):
- clarifying the mathematical formulation, eg what forms of $k_H$ are we examining, provide a full probabilistic model summary of the model, point out design choices
- pointing out differences or similarities with existing work
- remove gratuitous reference to deep learning in intro (it detracts)
- make sure that all important questions a reader might have are addressed
# Overall assessment
The issue addressed (modelling univariate outputs which were generated under different, known conditions) and the modelling choice (representing conditions as latent variables) are interesting.
I see no reason why to even mention and work with the (restrictive) parameterization of section 2.2, and not immediately the parameterization allowing "missing data" section 4. The latter is more interesting and equally simple.
Overall, the paper is reasonably clear, but seems lacking in originality and novelty. 
## Novelty
Wrt work on multi-output / multi-task GP, I can see that:
- this model has latent variables, not task IDs, and so can generalize over tasks (called conditions in the paper)
- the computational treatment using inducing points helps
The two seem very close though. The assumption of Kronecker structure is a staple. In particular, experiments should try compare to these methods (eg considering every condition a distinct task). 
The application of Titsias 2009 seems straightforward, I cannot see the novelty there, nor why so much space is used to present it.
## Presentation and clarity
1. Paper requires English proofreading. I usually report typos and mistakes in my reviews, but here there are too many grammar errors.
2. Motivating example with braking distance is helpful.
2. I don't understand fig 1 c,d at all: why do we not see the data points of fig 1a? where do these new points come from? what is depicted in the top vs the bottom plot?
2. References: the citation of Bussas 2017 is wrong. The complete correct citation is more like: Matthias Bussas, Christoph Sawade, Tobias Scheffer, and Niels Landwehr. Varying-coefficient models for geospatial transfer learning.. Machine Learning, doi:10.1007/s10994-017-5639-3, 2017.