Mind the Nuisance: Gaussian Process Classification using Privileged Noise
The learning with privileged information setting has recently attracted a lot of attention within the machine learning community, as it allows the integration of additional knowledge into the training process of a classifier, even when this comes in the form of a data modality that is not available at test time. Here, we show that privileged information can naturally be treated as noise in the latent function of a Gaussian process classifier (GPC). That is, in contrast to the standard GPC setting, the latent function is not just a nuisance but a feature: it becomes a natural measure of confidence about the training data by modulating the slope of the GPC probit likelihood function. Extensive experiments on public datasets show that the proposed GPC method using privileged noise, called GPC+, improves over a standard GPC without privileged knowledge, and also over the current state-of-the-art SVM-based method, SVM+. Moreover, we show that advanced neural networks and deep learning methods can be compressed as privileged information.
The paper suggests a new way for incorporating privileged data to the GPC model. The privileged data is used as a way to estimate the noise level of each input location. Both the noise level g and class probability f are learned jointly using EP with a one dimensional quadrature estimation. In my view this is an elegant way of using GPC with privileged information. If I understand correctly, adding function g() acts as a way to use locations with similar privileged information to better estimate the “noise” in that location.Quality: The paper is theoretically sound as it extends the GPC framework using an additive GP model. Overall the paper shows a good theoretical background and compares the results of several examples using different datasets. I would also like for the authors to mention something about the runtime complexity of GPC+.Clarity : Overall, the paper is clearly written. There are a few locations where editing of very long run-on sentences can help. There is also a italic run-away at line 38. The authors should spend less space explaining known ideas such as GPC and EP and explain more about the meaning of g(), e.g., what exactly does it mean hard/simple point, intuition using a simple example. Another interesting question is will the introduction of g actually hurts the model (as seen in a few examples in the results). Another important point that felt a bit swept under the rug was the approximation of Eq. 14. Why does this approximation work, and how well? Even if there is no room in the manuscript this should be discussed in the supplemental.Originality: The paper is original in the way it uses privilege information as the “noise” in the GPC model. Significance The paper showed improvements in several problems using very different datasets. My concern is that the improvement in accuracy might not justify the big increase in complexity of using the GPC+ framework. However, the idea behind GPC+ is nice and can allow a new perspective on the problem.