ATTENTIVE NEURAL PROCESSES
Neural Processes (NPs) (Garnelo et al., 2018a;b) approach regression by learning to map a context set of observed input-output pairs to a distribution over regression functions. Each function models the distribution of the output given an input, conditioned on the context. NPs have the benefit of fitting observed data efficiently with linear complexity in the number of context input-output pairs, and can learn a wide family of conditional distributions; they learn predictive distributions conditioned on context sets of arbitrary size. Nonetheless, we show that NPs suffer a fundamental drawback of underfitting, giving inaccurate predictions at the inputs of the observed data they condition on. We address this issue by incorporating attention into NPs, allowing each input location to attend to the relevant context points for the prediction. We show that this greatly improves the accuracy of predictions, results in noticeably faster training, and expands the range of functions that can be modelled.
Neural process (NP) is a recent probablistic method for modeling distributions of functions. The authors claim that one substantial weakness of NP is the tendency of under-fitting. The authors give a hypoethesize: the under-fitting behaviour of NP is because the mean-aggregation step in the encoder acts as a bottleneck, as a result, it is difficult for the decoder to learn the relevant information for a give target prediction. This paper proposes to resolve this issue by adding an attention mechanism to the deterministic path. The experimental results show that the proposed method converge faster and give better results on various tasks.
One major concern about the paper is the lack of analysis of the true cause of under-fitting in NP. The authors give the hypoethesize about the potential cause of the under-fitting issue and proposes to resolve it with attention, however, without theoretical or empirical analyses, it is hard to understand the true cause of the under-fitting issue. Although the proposed method give better performance, it is not clear whether the better performance is due to the added complexity to the model (the attention mechanism) or truely resolving the under-fitting issue. Some analyses along this line can make the paper clearer and more convincing.
A lot of technical details are missing in the paper, which makes the method not reproducible. Please add more details about the proposed attention mechanism and how they are implemented into NP.
In the GP literature, there are also methods tackling meta-learning or multi-task learning or few shot learning. These works are known as multi-output / multi-tasks Gaussian processes. A few works on this topic are listed:
* Z Dai, MA Álvarez, ND Lawrence, Efficient Modeling of Latent Information in Supervised Learning using Gaussian Processes, NIPS 2017
* MA Alvarez, L Rosasco, ND Lawrence, Kernels for vector-valued functions: A review, Foundations and Trends® in Machine Learning 2012
* EV Bonilla, KM Chai, C Williams, Multi-task Gaussian process prediction, NIPS 2007
For the 1D regression experiments (Figure 1left, Figure 3right), it is not clear which fitting is better. It largely depends on the prior of kerel parameters. As the data points are generated from a GP, plotting the Gaussian process fit with the ground truth parameters can show what a ground truth fitting would look like.
The Bayesian optimization experiment is very nice and gives some good insights about the quality of the uncertainty of prediction. Maybe consider it to include it in the main text.