summaries: Neural processes (NPs) aim to stochastically complete unseen data points based on a given context dataset. This paper incorporates the stochastic attention in NPs to capture the context information. We empirically show that our approach on 1D regression, predator-prey model, image completion, and MovieLens-10k dataset.

summaries: The paper proposes a novel method for the Attentive Neural Processes paradigm by adding stochasticity to the weights of the cross-attention module between the context and target representations. These weights are drawn from a proposal distribution which is a Weibull distribution with parameters determined by the context and target input points. A new KL regularization term is added to the total loss to enforce the proposal distribution be close to the gamma distribution determined only by the context points, in a similar fashion to the regularization term for the latent representation between the whole set of target and context points and that of the context points only. The authors include, from Fan’s paper, the closed-form KL divergence between the Weibull and Gamma distributions, making the whole loss be differentiable w.r.t. the network parameters thanks to the reparametrization trick, as in the original NP paper. The new loss is linked to the gain of the target distribution by means of information theory, by making the assumption that the new regularization implies not only that the mutual information between the target distribution and the target points is maximized, but also the mutual information between the representation for the context and target points and the latent representation of the context variables only is minimized, enforcing the latent variable to also consider the context points, and not otherwise as suggested by the original NP formulation.

summaries: This paper proposes to improve attentive neural processes (ANP; Kim et al.) by replacing deterministic attention with a Bayesian attention module. Since the inference requires variational approximation, I think the proposed method is a variational counterpart of ANP, similar to the relation between VAEs and autoencoders.

summaries: This paper proposes a neural process enhanced with stochastic attention to focus more on the context dataset. The method replace the classical attention used in ANP with the Bayesian attention module, showing that this design choice improves the performance also in noisy scenarios or when target datasets mismatched (by changing the kernel used to generate the target dataset). Moreover, the paper offers an interpretation of the method from an information theory perspective, proving that the NP with stochastic attention can be seen as a regularization of the latent space such that it pays more attention to the context dataset. The method is tested on both synthetic and real-world datasets and improves the scores especially in noisy or more complicated scenarios.

abstractText: Neural processes (NPs) aim to stochastically complete unseen data points based on a given context dataset. NPs essentially leverage a given dataset as a context representation to derive a suitable identifier for a novel task. To improve the prediction accuracy, many variants of NPs have investigated context embedding approaches that generally design novel network architectures and aggregation functions satisfying permutation invariant. In this work, we propose a stochastic attention mechanism for NPs to capture appropriate context information. From the perspective of information theory, we demonstrate that the proposed method encourages context embedding to be differentiated from a target dataset, allowing NPs to consider features in a target dataset and context embedding independently. We observe that the proposed method can appropriately capture context embedding even under noisy data sets and restricted task distributions, where typical NPs suffer from a lack of context embeddings. We empirically show that our approach substantially outperforms conventional NPs in various domains through 1D regression, predator-prey model, and image completion. Moreover, the proposed method is also validated by MovieLens-10k dataset, a real-world problem.

authors:
Alessandro Achille
Stefano Soatto

title: Information dropout: Learning optimal representations through noisy computation

venue: IEEE transactions on pattern analysis and machine intelligence,

authors:
Hareesh Bahuleyan
Lili Mou
Olga Vechtomova
Pascal Poupart

title: Variational attention for sequence-to-sequence models

venue: In Proceedings of the 27th International Conference on Computational Linguistics,

title: Computing the kullback-leibler divergence between two generalized gamma distributions

venue: arXiv preprint arXiv:1401.6853,

authors:
Rianne van den Berg
Thomas N Kipf
Max Welling

title: Graph convolutional matrix completion

venue: arXiv preprint arXiv:1706.02263,

authors:
Yuri Burda
Roger B Grosse
Ruslan Salakhutdinov

title: Importance weighted autoencoders

venue: In ICLR (Poster),

authors:
Yuntian Deng
Yoon Kim
Justin Chiu
Demi Guo
Alexander Rush

title: Latent alignment and variational attention

venue: Advances in Neural Information Processing Systems,

authors:
Harrison Edwards
Amos Storkey

title: Towards a neural statistician

venue: In International Conference on Learning Representations,

authors:
Xinjie Fan
Shujian Zhang
Bo Chen
Mingyuan Zhou

title: Bayesian attention modules

authors:
Andrew Foong
Wessel Bruinsma
Jonathan Gordon
Yann Dubois
James Requeima
Richard Turner

title: Meta-learning stationary stochastic process prediction with convolutional neural processes

authors:
Alexandre Galashov
Jonathan Schwarz
Hyunjik Kim
Marta Garnelo
David Saxton
Pushmeet Kohli
SM Eslami
Yee Whye Teh

title: Meta-learning surrogate models for sequential decision making

authors:
Marta Garnelo
Dan Rosenbaum
Christopher Maddison
Tiago Ramalho
David Saxton
Murray Shanahan
Yee Whye Teh
Danilo Rezende
SM Ali Eslami

title: Conditional neural processes

authors:
Jonathan Gordon
Wessel P Bruinsma
Andrew YK Foong
James Requeima
Yann Dubois
Richard E Turner

title: Convolutional conditional neural processes

authors:
Soyeon Caren Han
Taejun Lim
Siqu Long
Bernd Burgstaller
Josiah Poon

title: Glocal-k: Global and local kernels for recommender systems

venue: arXiv preprint arXiv:2108.12184,

authors:
Hyunjik Kim
Andriy Mnih
Jonathan Schwarz
Marta Garnelo
Ali Eslami
Dan Rosenbaum
Oriol Vinyals
Yee Whye Teh

title: Attentive neural processes

authors:
Diederik P Kingma
Jimmy Ba

title: Adam: A method for stochastic optimization

authors:
Durk P Kingma
Tim Salimans
Max Welling

title: Variational dropout and the local reparameterization trick

venue: In Advances in neural information processing systems,

authors:
Dieterich Lawson
Chung-Cheng Chiu
George Tucker
Colin Raffel
Kevin Swersky
Navdeep Jaitly

title: Learning hard alignments with variational inference

venue: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP),

authors:
Juho Lee
Yoonho Lee
Jungtaek Kim
Eunho Yang
Sung Ju Hwang
Yee Whye Teh

title: Bootstrapping neural processes

authors:
Janarthanan Rajendran
Alexander Irpan
Eric Jang

title: Meta-learning requires meta-augmentation

venue: In Advances in Neural Information Processing Systems,

authors:
Ahmed Rashed
Josif Grabocka
Lars Schmidt-Thieme

title: Attribute-aware non-linear coembeddings of graph features

venue: In Proceedings of the 13th ACM Conference on Recommender Systems,

authors:
Andrei A Rusu
Dushyant Rao
Jakub Sygnowski
Oriol Vinyals
Razvan Pascanu
Simon Osindero
Raia Hadsell

title: Meta-learning with latent embedding optimization

authors:
Shiv Shankar
Sunita Sarawagi

title: Posterior attention models for sequence to sequence learning

authors:
Gautam Singh
Jaesik Yoon
Youngsung Son
Sungjin Ahn

title: Sequential neural processes

title: A generalization of the gamma distribution

venue: The Annals of mathematical statistics,

authors:
Ashish Vaswani
Noam Shazeer
Niki Parmar
Jakob Uszkoreit
Llion Jones
Aidan N Gomez
Łukasz Kaiser
Illia Polosukhin

title: Attention is all you need

authors:
Michael Volpp
Fabian Flürenbrock
Lukas Grossberger
Christian Daniel
Gerhard Neumann

title: Bayesian context aggregation for neural processes

title: Stochastic modelling for systems biology

authors:
Jin Xu
Jean-Francois Ton
Hyunjik Kim
Adam Kosiorek
Yee Whye Teh

title: Metafun: Metalearning with iterative functional updates

authors:
Jaesik Yoon
Gautam Singh
Sungjin Ahn

title: Robustifying sequential neural processes

authors:
Hao Zhang
Bo Chen
Dandan Guo
Mingyuan Zhou

title: Whai: Weibull hybrid autoencoding inference for deep topic modeling

authors:
2018a Garnelo et al
2019 Kim et al
Lee

heading: 1 INTRODUCTION

text:
Neural processes (NPs) have been in the spotlight as they stochastically complete unseen target points considering a given context dataset without huge inference computation (Garnelo et al., 2018a;b; Kim et al., 2019). NPs leverage neural networks to derive an identifier suitable for a novel task using context representation, which contains information about given context data points. These methods enable us to handle considerable amounts of data points, such as in image-based applications, that the Gaussian process cannot naively deal with.
Many studies have revealed that the prediction performance relies on the way of context representation (Kim et al., 2019; Volpp et al., 2021). The variants of NPs have mainly investigated on context embedding approaches that generally design novel network architectures and aggregation functions that are permutation invariant. For example, Garnelo et al. (2018a;b) developed NPs using MLP layers to embed context set via mean aggregation. Bayesian aggregation satisfying permutation invariance enhanced the prediction accuracy by varying weights for individual context data points (Volpp et al., 2021). From the prospective of network architectures to increase model capacity, Kim et al. (2019) suggested the way of local representation to consider relations between context dataset and target dataset using deterministic attention mechanism (Vaswani et al., 2017). For robustness under noisy situations, bootstrapping method was proposed to orthogonally apply to variants of NPs (Lee et al., 2020).
Despite many appealing approaches, one significant drawback of previous NPs is that they still underfit when confronted with noisy situations like real-world problems. This manifests as inaccurate predictions at the locations of the context set as seen in Figure 1a. Additionally, in Figure 1b, the attentive neural process Kim et al. (2019) fails to capture contextual embeddings because the attention weights of all target points highlight on the lowest value or the maximum value in the context dataset. In the case of the Bootstrapping ANP Lee et al. (2020) and ANP with information dropout, the quality of heat-map is slightly improved, but it still falls short of ours. This indicates that the present NPs are unable to properly exploit context embeddings because the noisy situations impair the learning of the context embeddings during meta-training.
To address this issue, we propose a newly designed neural process to fundamentally improve performance by paying more attention to the context dataset. The proposed method expedites a stochastic attention to adequately capture the dependency of the context and target datasets by adjusting stochasticity. It results in improving prediction performance due to maintained context information. We observe that our proposed algorithm works well by utilizing contextual information in an intended manner as seen in Figure 1. This method outperforms current NPs and their regularization methods in all experiment settings. Thus, this paper clarifies the proposed method as regularized NPs in terms of information theory, explaining that the stochastic attention encourages the context embedding to be differentiated from the target dataset. This differentiated information induces NPs to become appropriate identifiers for target dataset by paying more attention to context representation. To summarize, we make the following contributions:
• We propose the novel neural process that pay more attention to the context dataset. We claims for the first time, using the information theory framework, that critical conditions for contextual embeddings in NPs are independent of target features and close to contextual datasets. • Through comprehensive analyses, we illustrate how stochastic local embeddings are crucial
for NPs to focus on capturing the dependencies of context and target datasets. Even when context dataset contains noise or is somewhat different from target dataset, as shown in Lee et al. (2020), the proposed method is capable of adapting a novel task while preserving predictive performance. Particularly, this method significantly enhances performance without additional architectures and data augmentation compared to the attentive neural process (Kim et al., 2019). • The experimental results show that the proposed model substantially outperforms conven-
tional NPs in typical meta-regression problems. For instance, the proposed method achieves to obtain the state of the art score in the image completion task with the CelebA dataset. Especially, the proposed model maintains performance in the limited task distribution regimes such as the MovieLenz-10k dataset with a small number of users.
This paper is organized as follows. We introduce background knowledge in section 2. section 3 presents a neural process with stochastic attention and related works illustrated in section 5. Experimental results are shown in section 4, concluding with section 6.

heading: 2 BACKGROUND

heading: 2.1 NEURAL PROCESSES

text:
Suppose that we have an observation set X = {xi}ni=1 and a label set Y = {yi} n i=1. NPs (Garnelo et al., 2018a;b; Kim et al., 2019) are designed to obtain the probabilistic mapping from the observation
set to the label set p(Y |X,Xc, Yc) given a small subset (Xc, Yc) = (xj , yj)mj=1. Basically, it is built upon the neural network with an encoder-decoder architecture where the encoder fφ outputs a task representation by feed-forwarding (Xc, Yc) through permutation-invariant set encoding (Zaheer et al., 2017; Edwards & Storkey, 2017) and the decoder fθ models the distribution of Y (e.g Gaussian case : estimating µ, σ) using X along with the encoder outputs. Its objective is to maximize the log likelihood over the (unknown) task distribution p(T ). All tasks provided by the data generating process are considered as Monte Carlo samples, respectively:∑
Tk∼p(T )
log p(Y |X,Xc, Yc) = ∑
Tk∼p(T )
n∑ i=1 logN (yi|µi, σi)
where (µi, σi) = fθ(xi, r), r = fφ({xj , yj}mj=1) (1)
The set encoding architecture differentiates the type of NPs. In Conditional neural process(CNP) (Garnelo et al., 2018a), the representation r is a deterministic variable such that, by using mean aggregation, the encoder maps the context set into a single deterministic representation r = 1/m ∑m j=1 fφ(xj , yj). Neural process (NP) (Garnelo et al., 2018b) introduces a probabilistic latent variable z to model functional uncertainty as a stochastic process such that the parameters of output distribution may change according to the sampled value of z. Due to the intractable log-likelihood, a training objective is derived based on variational inference which can be decomposed into two terms, reconstruction term and regularization term:
log p(Y |X,Xc, Yc) ≥ Eqφ(z|X,Y ) [ log
pθ(Y |X, r, z)pθ(z|Xc, Yc) qφ(z|X,Y ) ] = Eqφ(z|X,Y ) [log pθ(Y |X, r, z)]− KL(qφ(z|X,Y )||pθ(z|Xc, Yc)) (2)
For simplicity, the prior distribution pθ(z|Xc, Yc) is approximated by qφ(z|Xc, Yc). However, as pointed out in Kim et al. (2019), the mean aggregation over the context set is too restrictive to describe the dependencies of the set elements. To enhance the expressiveness of the task representation, Attentive neural process(ANP) accommodates an attention mechanism (Vaswani et al., 2017) into the encoder, which generates the local deterministic representation ri corresponding to a target data point xi, and addresses the underfitting issue in NP. Although additional set encoding methods such as the kernel method and the bayesian aggregation considering task information have been suggested (Xu et al., 2020; Volpp et al., 2021), the attentive neural process is mainly considered as the baseline in terms of model versatility.

heading: 2.2 BAYESIAN ATTENTION MODULE

text:
Consider m key-value pairs, packed into a key matrix K ∈ Rm×dk and a value matrix V ∈ Rm×dv and n queries packed into Q ∈ Rn×dk , where the dimensions of the queries and keys are the same. Attention mechanisms aim to create the appropriate values O ∈ Rn×dv corresponding to Q based on the similarity metric to K, which are typically computed via an alignment score function g such that Φ = g(Q,K). Then, a softmax function is applied to allow the attention weight W ∈ Rn×m to satisfy the simplex constraint so that the output features can be obtained by Oi,j = Wi · Vj :
Wi,j = exp(Φi,j)∑m j=1 exp(Φi,j)
(3)
Note that there are many options for the alignment score function g where a scaled dot-product or a neural network are widely used. Bayesian attention module (Fan et al., 2020) considers a stochastic attention weight W . Compared to other stochastic attention methods (Shankar & Sarawagi, 2018; Lawson et al., 2018; Bahuleyan et al., 2018; Deng et al., 2018), it requires minimal modification to the deterministic attention mechanism described above so that it is compatible with the existing frameworks, which can be adopted in a straightforward manner. Specifically, un-normalized attention weights Ŵ such that Wi,j = Ŵi,j/ ∑m j=1 Ŵi,j are sampled via the variational distribution qφ(Ŵ |Q,K), which can be trained via amortized variational inference:
log p(O|Q,K, V ) ≥ Eqφ(Ŵ |Q,K) [ log pθ(O|V, Ŵ ) ] − KL(qφ(Ŵ |Q,K)||pθ(Ŵ )) (4)
Considering the random variable Ŵ to be non-negative such that it can satisfy the simplex constraint by normalization, the variational distribution qφ(Ŵ |Q,K) is set to Weibull(k, λ) and the prior distribution pθ(Ŵ ) is set to Gamma(α, β). This λ can be obtained by the standard attention mechanism, on the other hands, α can be either a learnable parameter or a hyper-parameter depending on that the model follows key-based contextual prior described in the following paragraph. The remaining variables k and β are regarded as user-defined hyper-parameters. By introducing Euler–Mascheroni constant γ (Zhang et al., 2018; Bauckhage, 2014), the KL divergence in Equation 4 can be computed in an analytical expression (shown in Appendix A).
KL(Weibull(k, λ)||Gamma(α, β)) = γα k −α log k+βλΓ(1+ 1 k )−γ−1−α log β+ logΓ(α) (5)
Samples from Weibull distribution can be obtained using a reparameterization trick exploiting an inverse CDF method: λ(− log(1 − )) 1k , ∼ Uniform(0, 1). Note that mean and variance of Weibull(k, λ) are computed as λΓ(1 + 1/k) and λ2 [ Γ(1 + 2/k)− (Γ(1 + 1/k))2 ] . It can be observed that the variance of obtained samples decreases as k increases.
Key-based contextual prior To model the prior distribution of attention weights, key-based contextual prior was proposed. This method allows the neural network to calculate the shape parameter α of the gamma distribution as a prior distribution. This leads to the stabilization of KL divergence between standard attention weights and sampled attention weights, and it prevents overfitting of the attention weights (Fan et al., 2020). In this paper, we explain the reason why the key-based context prior is important for capturing an appropriate representation focusing on a context dataset from the prosepective of information theory.

heading: 3 NEURAL PROCESSES WITH STOCHASTIC ATTENTION

text:
We expect NPs to appropriately represent the context dataset to predict target data points in a novel task. However, if the context dataset has a limited task distribution and noise like real-world problems, conventional NPs tend to sensitively react to this noise and maximize the objective function including irreducible noise. These irreducible noises do not completely correlate with context information, so that this phenomenon derives meaningless set encoding of the context dataset in training phase and hinders adaptation to new tasks. In other words, the output of NPs pθ(yi|xi, z, ri) does not depend on (z, ri).
To preserve the quality of context representation, we propose a method that better utilizes the context dataset by exploiting stochastic attention with the key-based contextual prior to NPs. We show that the proposed method enables to create more precise context encoding by adjusting stochasticity, even in very restricted task distribution regimes.

heading: 3.1 GENERATIVE PROCESS

text:
As with the Attentive neural process(Kim et al., 2019), the proposed method consists of the two types of encoders and a single decoder architecture. The first encoder embeds (Xc, Yc) to a global representation z and the second encoder makes a local representation ri by (xi, Xc, Yc). The global representation z serves to represent entire context data points (Xc, Yc), whereas the local representation ri is in charge of fine-grained information between target data xi and context (Xc, Yc) for predicting the output distribution p(yi|xi, (z, ri)). Unlike the ANP, the proposed model considers all intermediate representations (z, ri) as stochastic variables.
We exploit the Bayesian attention module to create a local representation ri by the stochastic attention weights wi with the key-based contextual prior (Fan et al., 2020). To use the reparameterization trick for sampling stochastic variables, we draw random noise samples 1 ∼ Unif(0, 1), 2 ∼ N (0, 1). First, as mentioned in section 2, the stochastic attention weights
{wi}Ni=1 are obtained via the inverse CDF of the Weibull distribution with random noise 1. For amortized variational inference, the prior distribution qφ({w}Ni=1|K(Xc)) is also derived with context dataset Xc for implementing key-based contextual prior. Meanwhile, 2 is used to obtain the global representation z sampled from normal distribution. The entire scheme is shown in Figure D.2c. As shown in section 2, we can calculate all the KL divergences of z and {w}ni=1 as closed-form solutions. The decoder follows the standard neural processes p(yi|xi, z, ri) = N (yi|µφ(xi, z, ri), σ2φ(xi, z, ri)). See Appendix D for implementation details.

heading: 3.2 LEARNING AND INFERENCE

text:
Unlike the Attentive neural process(Kim et al., 2019), the proposed method regards all representations z and W = {wi}ni=1 as stochastic variables, so that we clearly drive the objective function according to amortized variational inference. Based on the objective function of Neural process(Garnelo et al., 2018b), the proposed method adds KL divergence of stochastic attention weight W = {wi}ni=1. Assuming the independence between z and {wi}ni such that individual wi is only dependent on xi and (Xc, Yc) as seen in Figure 2, the objective function for each task Tk is presented as follows:
LTk(φ, θ) = N∑ i=1 [ log pθ(yi|xi, z, ri)− KL ( qφ(wi|xi, Xc)|qφ(wi|Xc) )] − KL ( qφ(z|X,Y )|qφ(z|Xc, Yc)
) (6) Note that (X,Y ) follows a task Tk and each task Tk is drawn from task distribution p(T ). The final objective function is L(φ, θ) = ETk [LTk(φ, θ)]. From the perspective of amortized variational inference, the prior distributions of p(z) and p({wi})ni=1 should be defined. With regard to z, we follow the standard neural processes wherein p(z) is defined as qφ(z|{Xc, Yc} (Garnelo et al., 2018b). In the case of {wi}Ni=1, we introduce the strategy of Bayesian attention modules (Fan et al., 2020). The prior distribution of {wi}Ni=1 defined as a key-based contextual prior, qφ(wi|Xc), not only stabilizes the KL divergence but also activates the representation wi to pay more attention to the context dataset. In the next section, this objective function can be described in terms of information theory as role of regularization to pursue the original goal of NPs, which is deriving appropriate identifiers for target datasets in novel tasks.

heading: 3.3 DISCUSSION OF OBJECTIVE FUNCTION FROM VIEWS OF INFORMATION THEORY

text:
A novel part of the proposed method is the use of the stochastic attention mechanism to more leverage context information, thereby stably capturing the dependency of context and target datasets in noisy and very limited task distributions. In this subsection, based on information theory, we elaborate the goal of NPs and our novel stochastic attention mechanism explained as the regularization of latent variables to pay more attention to context datasets. This enables us to understand latent variables and their requirements semantically. According to subsection 3.2, the objective function is categorized into two terms: a reconstruction log pθ(yi|xi, z, ri) and regularization as two KL Divergences. Suppose that {xi, yi} is a data point in a target dataset, D is a given context dataset, latent variables z and ri are considered as Z, which is defined as the information bottleneck forD, we suggest that maximizing a reconstruction term corresponds to increase I(yi,D|xi) and minimizing two KL Divergence means to decrease I(Z, xi|D). First, the mutual information of the target value yi and the context dataset D given the input feature xi, I(yi,D|xi), is the metric to identify that NPs is adapted to a novel task. Suppose p(yi|xi,D) is output value of NPs and p(yi|xi) = ED [ p(yi|xi,D) ] is output value not conditioned on context dataset D. If NPs completely fail to adapt new tasks using the context dataset D, the suggested metric I(yi,D|xi) goes to 0.
I(yi,D|xi) = Eyi,D [ log
pθ(yi|xi,D) pθ(yi|xi)
] (7)
Assumed that target data point {xi, yi} is sampled from Tk, which is uncontrollable data generating process, the reconstruction term ∑ {xi,yi} log p(yi|xi, Z) can be regarded to I(yi;Z|xi) based on
information bottleneck theorem; I(yi;D|xi) ≥ I(yi;Z|xi) holds. A detailed explanation is given in Appendix C. The objective function should be designed to increase I(yi,D|xi) by an appropriate identifier for a novel task considering a context dataset D. However, as mentioned in the previous sections, NPs suffer from irreducible noises and limited task distribution in meta training. This phenomena means that NPs increase the objective function by learning the way to directly map from xi to yi including noises and some features only in meta-training, instead of taking into consideration of the contextD. To make latent variablesZ capture proper dependencies, we define the regularization accounting for latent variables to pay more attention to the context dataset.
This regularization is the mutual information of latent variable Z and input xi given context dataset D to indicates how similar information latent variable Z and target xi contain. We regard p(Z|xi, D) as the distribution of latent variables that considers both context dataset D and target xi, meanwhile p(Z|D) = Exi [ p(Z|xi, D) ] as the distribution of latent variables only depending on the context dataset D. If latent variables Z have totally different information against the target xi given the context dataset D, It can be I(Z, xi|D) = 0
I(Z, xi|D) = EZ,xi [ log
p(Z|xi,D) p(Z|D)
] (8)
We bring I(yi;D|xi)− I(Z, xi|D) to be maximized instead of I(yi,D|xi) so as to model adequately dependency between context and the target dataset. By decreasing I(Z, xi|D) and increasing I(yi,D|xi) in meta-training, we expect that the model learns the way to differentiate Z and xi and construct an identifier which can further consider Z and xi together. Note that φ is the parameter of the function to transform D to the information bottleneck Z and θ is the parameter of the identifier for target data point yi, we show that I(yi;D|xi) − I(Z, xi|D) ≥ Iφ,θ(yi;Z|xi) − Iφ(Z, xi|D) if φ, θ is perfectly able to leverage all information about Z and xi. We assume that neural network architectures can perform an intended manners.
Theorem 1. Let Z be the representation of the context dataset D and it follows an information bottleneck. The following equation holds when the latent variable Z can be split into {z, wi} and z is only dependent on D, but wi is dependent on D and xi,
LTk(φ, θ) ≤ I(yi;D|xi)− I(Z, xi|D) (9)
where wi is obtained by the Bayesian attention modules (Fan et al., 2020), and LTk is defined in Equation 6 and its probabilistic graphical model follows Figure 2.
As mentioned, if the neural network architectures φ, θ are able to completely represent to Z and yi given D and xi, LTk(φ, θ) = Iφ,θ(yi;Z|xi)− Iφ(Z;xi|D) ideally holds. In this subsection, we reveal that stochastic attention mechanism satisfy on this regularization as mentioned. p(Z|xi,D) is regarded as qφ(wi|xi, {Xc, Yc}) and p(Z|D) is the key-based contextual prior qφ(wi|K(Xc)). Therefore, the stochastic attention mechanism with key-prior that requires D and xi as well as being able to this regularization for latent variables to pay more attention to the context dataset in Equation 8 is considered. By Theorem 1, the network parameters φ and θ are updated by the gradient of the objective function in Equation 6 semantically improves I(yi;D|xi) − I(Z, xi|D). Therefore, we present the experimental results in the next section. They indicate that we can implement a model that can complete unseen data points in novel tasks by fully considering the context representations. The detailed analysis is presented in Appendix G and Appendix H.

heading: 4 EXPERIMENT

text:
In this section, we describe our experiments to answer three key questions: 1) Are existing regularization methods such as weight decay, importance-weighted ELBO as well as recent methods(Information dropout, Bootstrapping, and Bayesian aggregation) effective to properly create context representations ?; 2) Can the proposed method reliably capture dependencies between target and context datasets even under noisy situations and limited task distributions ?; 3) Is it possible to improve performance via the appropriate representation of the dataset ?
To demonstrate the need for neural process with stochastic attention, we choose CNP, NP (Garnelo et al., 2018a;b), and ANP (Kim et al., 2019) as baselines, which are commonly used. For all baselines,
we employ weight decay and importance-weighted ELBO that requires several samples against one data point and aggregated by log ∑ i expLk. Recent regularization methods can be also considered. Information dropout(Kingma et al., 2015; Achille & Soatto, 2018), Bayesian aggregation (Volpp et al., 2021) and Bootstrapping (Lee et al., 2020) are chosen. Bayesian aggregation is used for CNP and NP according to the original paper, meanwhile Information dropout and Bootstrapping are used for the ANP to verify regularization of local representations. When training, all methods follow their own learninig policy; however, for fair comparison, all methods perform one Monte-Carlo sampling across a test data point during testing. We respectively conduct 5 trials for each experiment and report average scores. As a metric to evaluate the model’s predictive performance, we use the likelihood of the target dataset, while the likelihood of the context dataset is considered as a metric to measure the extent to which models can represent a context dataset. Since these metrics are proportional to model performance, a higher value indicates a better performance. In these experiments, we show that the proposed method substantially outperforms various NPs and their regularization methods.

heading: 4.1 1D REGRESSION UNDER NOISY SITUATIONS

text: We conduct synthetic 1D regression experiments to test whether our model adequately captures the dependency of a dataset under noisy situation. We trained all models with samples from the RBF GP functions. We tested them with samples from various kernel functions (Matern, Periodic inclduing RBF) to identify that models is capable of adaptatation to new tasks. To consider noisy environments, we artificially generate noises into the training dataset. Our model maintains outstanding performance compared to other baselines. In particular, when the test data comes from periodic kernel GP, which is the most different from the RBF GP, the proposed model utilizes context datasets appropriately regardless of the test data sources, whereas other models do not properly use the context data. When comparing the likelihood values of context datasets, the proposed method preserves equivalent scores in all cases, but the others do not. As shown in Figure 1a, the proposed method captures context points better than other methods, as only it method correctly captures the relationship even under noisy situations. These results support our claim, the proposed method utilizes context datasets in prediction. We report graphical explanation and the additional experimental result of 1D regression without noises in Appendix E.

heading: 4.2 PREDATOR-PREY MODEL AND IMAGE COMPLETION

text:
We apply NPs to a domain shift problem called the predator-prey model, proposed in the ConvCNP(Gordon et al., 2019). In this experiment, we train models with the simulated data and applied
models to predict the real-world data set, Hudson’s bay the hare-lynx. Note that the hare-lynx dataset is very noisy, unlike the simulated data. The detailed explanation of datasets is written in Appendix F. Recent regularization methods such as Bayesian aggregation, Information dropout, and Bootstrapping are effective to improve performance. In particular, Bootstrapping method significantly influences predictive performance compared of other ANPs as mentioned in the original paper(Lee et al., 2020). However, we observe that the proposed method is superior to the other models in both simulated test set and real-world dataset. In particular, all baselines drastically decrease the likelihood values due to noises during the test, while the proposed model preserves the performance because it is robust to noises. See the left side of Table 2. We report the additional experimental result with periodic noises and the graphical explanation is presented in Appendix F.
Second, we conduct the image completion task in which the models generate images under some given pixels. The experiment setting follows the previous experiments (Garnelo et al., 2018a;b; Kim et al., 2019; Lee et al., 2020). To conduct fair comparison, all models employ only MLP encoders and decoders and the multi-head cross attention is used for local representations; the variant of ANPs. We indicate that the proposed method records the best score as seen in the right side of Table 2. The best baseline records 3.53 with 500 context pixels in Table 2, whereas our method attains 2.653 of likelihood even with 50 context pixels and grows to 3.948 with 500 context pixels. Referred to Lee et al. (2020)’s paper, the previous highest score is 4.150 of likelihood for context pixels and 3.129 of likelihood for target pixels. As Kim et al. (2019) mentioned that the more complex architecture like the self-attention mechanism enhances the completion quality than the MLP encoders, these scores were obtained by ANP with self-attention encoders and employing bootstrapping. However, we achieves to obtain comparable results for context points and exceed the previous highest score for target points by a significant margins without exhausted computations and complicated architectures. The completed images by ours and baselines are shown in Appendix J.

heading: 4.3 MOVIELENS-100K DATA

text:
We demonstrate the robustness and effectiveness of our method using a real-world dataset, the MovieLenz dataset, which is commonly used in recommendation systems. This setting has an expensive data collection process that restricts the number of users. In this section, we report how well the
proposed method can be generalized to novel tasks using limited tasks during the meta-training. To train NPs, we decide to split this dataset according to user ID and regard the rating samples made by one user as a task. The purpose of this experiment is that NP provides an identifier to serve for new visitor using very few rating samples. We follow the setting used in the existing work(Galashov et al., 2019). The proposed model performs better than the other methods. As mentioned in subsection 4.2, it captures the information from the context dataset, while other methods suffer from noise in the data and lack of users in meta-training. The experiment result of comaprison with baselines is reported in Appendix I. To validate use of real applications, we compare with existing studies.
According to Movielens-100k benchmarks, the state-of-the-art RMSE score is about 0.890, which can be obtained using graph structures about users and films(Rashed et al., 2019). For fair comparison, we train and test the proposed method on the same setting, named as U1 splits. When evaluate baselines on the u1.test dataset, we randomly draw samples of corresponding users from the u1.base used in the meta-training and regard as context data points. Although we do not use graph structures, as seen in Table 3, The ANP points the comparable result of 0.909, meanwhile the proposed model attains a promis-
ing result, 0.895 of the RMSE value. This experiment indicates that the proposed method can reliably adapt to new tasks even if it provides small histories, and we identify again that our model can properly work on noisy situations.

heading: 5 RELATED WORKS

text:
The stochastic attention mechanism enables the capturing of complicated dependencies and regularizing weights based on the user’s prior knowledge. However, such methods cannot utilize back-propagation because they do not consider the reparameterization trick to draw samples (Shankar & Sarawagi, 2018; Lawson et al., 2018; Bahuleyan et al., 2018; Deng et al., 2018). Even if these methods employ a normal distribution as a posterior distribution of latent variables, satisfying the simplex constraints, which sum to one (Bahuleyan et al., 2018), is impossible. Recently, the Bayesian attention module suggests that the attention weights are samples from the Weibull distribution whose parameters can be reparameterized, so this method can be stable for training and maintaining good scalability (Fan et al., 2020).
Since the Conditional neural process have been proposed(Garnelo et al., 2018a;b), several studies have been conducted to improve the neural processes in various aspects. The Attentive neural process modify the set encoder as an cross-attention mechanism to increase the performance of the predictability and interpretability (Kim et al., 2019). Some studies investigate NPs for sequential data (Yoon et al., 2020; Singh et al., 2019). Trials have been combined with optimization-based meta-learning to increase the performance of NPs for the image classification task with a pre-trained image encoder (Rusu et al., 2018; Xu et al., 2020). The convolutional network can be used for set encoding to obtain translation-invariant predictions specialized in image-based problems (Gordon et al., 2019; Foong et al., 2020). Similar to this study, the Bayesian context aggregation suggests the importance of context information over tasks(Volpp et al., 2021) and the Bootstrapping NPs orthogonally improves the predictive performance under the model-data mismatch. (Lee et al., 2020). Unfortunately, there is no clear explanation of how stochasticity has helped improve performance and does not show a fundamental approach to enhancing dependency between the context and the target dataset in terms of set encoding.

heading: 6 CONCLUSION

text: In this work, we propose an algorithm, neural processes with stochastic attention. We show that the proposed method can be explained from information theory. We devised the new regularization to lead representations paying attention to the context dataset and help our model pay more attention to the context dataset.

heading: A CLOSED FORM SOLUTION FOR THE KL DIVERGENCE BETWEEN WEIBULL AND GAMMA DISTRIBUTION

text:
The generalized gamma distribution contains both Weibull and gamma distribution as special cases. We are concerned with the three-parameter version of generalized gamma distribution introduced in Stacy (Stacy et al., 1962). Its parameters can be categorized into one scale parameter a and two shape parameters d and p. Its probability density function is defined for x ∈ [0,∞) and given by
f(x|a, d, p) = p ad
xd−1
Γ(d/p) exp
[ −(x a )p ]
(A.1)
where Γ(·) is the gamma function, and parameters a, d, p > 0. For d = p, it corresponds to the Weibull distribution, and if p = 1, it becomes the gamma distribution. Bauckhage (Bauckhage, 2014) derives a closed form solution for the kullback-leibler divergence between two generalized gamma distribution as follow.
KL(f1|f2) := ∫ ∞ 0 f1(x|a1, d1, p1) log f1(x|a1, d1, p1) f2(x|a2, d2, p2) dx
= log
( p1a p2 2 Γ(d2/p2)
p2a p1 1 Γ(d1/p1)
) + [ ψ(d1/p1)
p1 + log a1
] (d1 − d2)
+ Γ(d1+p2p1 )
Γ(d1p1 )
( a1 a2 )p2 − d1 p1
(A.2)
where f1, f2 are the generalized gamma distributions, and ψ(·) is digamma function. The Weiull distribution with scale parameter λ and shape parameter K coincides with the generalized gamma distribution a = λ and d, p = K. For gamma distribution with scale parameter β and shape parameter α, it becomes the generalized gamma distribution a = 1/β, d = α and p = 1. The KL divergence between the Weibull distribution (λ, k) and the gamma distribution (α, β) amounts to
KL(f1|f2) = log k − α log β + log Γ(α)
+ ψ(1)− α ks ψ(1)− α log λ+ λβΓ(1 + 1/K)− 1
(A.3)
We introduce ψ(1) = −γ ≈ 0.5772 as Euler constant. Finally, we obtain Equation 5
KL(Weibull(k, λ)||Gamma(α, β)) = γα k − α log λ+ log k + βλΓ
( 1 + 1
k ) − γ − 1− α log β + log Γ(α)
(A.4)

heading: B ELBO DERIVATIONS

text:
In this section, we derive the objective function in the manuscript of this paper. Without loss of generality, target dataset {X,Y } = {xi, yi}Ni=1 and context dataset D = (Xc, Yc) = {xi, yi}Mj=1 such that N M . Let the log-likelihood of target data points in a task Tk be log p(Y |X,D). We begin to maximizing the log-likelihood of target data points based on context dataset and representations. We suppose that it follows the graphical model in Figure D.2c
logp(Y |X,D) = log ∫ p(Y,Z|X,D)dz (B.5)
= log ∫ p(Y, Z|X,D)q(Z|X,D)
q(Z|X,D) dz (B.6)
= log ∫ p(Y,Z|X,D) q(Z|X,D) q(Z|X,D)dz (B.7)
= log ∫ ∏ i=1 p(yi|xi, Zi) q(Zi|xi,D) p(zi|xi,D)q(zi|xi,D)dz (B.8)
= logEq(z) [∏ i=1 p(yi|xi, Zi) q(Zi|xi,D) p(zi|xi,D) ]
(B.9)
Applying Jensen’s inequality to
log p(Y |X,D) ≥ Eq(z) [ log ∏ i=1 p(yi|xi, Zi)p(Zi|xi,D) q(zi|xi,D) ] (B.10)
≥ Eq(z) [∑ i=1 log pθ(yi|xi, Zi)− log q(Zi|xi,D) p(Zi|xi,D) ] (B.11)
From Zi = {z, wi} in Figure D.2c, we define z is dependent on context datasetD andwi is dependent on target xi and context dataset D
log p(Y |X,D) ≥ Eq(z) [∑ i=1 log pθ(yi|xi, Zi)− log q(wi|xi,D) p(wi|xi,D) − log q(z|D) p(z|D) ] (B.12)
We assume that p(wi|xi,D) ≈ qφ12 (wi|K(Xc)) by regularization of latent variables to pay attention on context dataset. It also follows the strategy of the standard neural processes : posterior distribution of z is qφ2(wi|{Xt, Yt}) and prior distribution of z is qφ2(wi|{Xc, Yc}). Thereby, we show that ELBO become the objective function of the proposed method in Equation 6.
log p(Y |X,D) ≥ Eq(z) [∑ i=1 log pθ(yi|xi, Zi)− log qφ11 (wi|xi,D) q12(wi|,K(Xc)) − log qφ2(z|{Xt, Yt}) qφ2(z|{Xc, Yc}) ] = LTk
(B.13)

heading: C PROOF OF THEOREM 1

text:
We present that I(yi;D|xi) has relation to the objective function of the proposed method. Suppose that Z is information bottleneck for D and all variables follow the graphical models Figure 2. As described in Equation 7, the mutual information I(yi,D|xi) is to measure dependency of yi and context dataset D. We should maximize this information to generalize the novel task. For simplicity, we set all latent variables as information bottleneck Z for the context dataset D. Note that Z = ({wi}ni=1, z) is information bottleneck corresponding to context representations in NPs, where W = {wi, . . . , wn} is an attention weights and X,Y = {(x1, y1), . . . , (xn, yn)} is respectively data points in the target data set. φ is regarded as the parameter of the function to transform D to the information bottleneck Z and θ is the parameter of the identifier for target data point yi. By information bottleneck theorem, Equation C.14 holds.
I(yi;D|xi) ≥ Iφ,θ(yi;Z|xi) = −Hφ,θ(y|xi, Z) +H(y|xi) (C.14)
The target data point {xi, yi} is drawn by the data generating process Tk ∼ p(T ), so that H(y|xi) can be regarded as a constant value due to uncontrollable factors. As a result, −Hφ,θ(y|xi, Z) is written as
−Hφ,θ(yi|xi, Z) = E(x,y) [ − log pθ(yi|xi, Z) ] (C.15)
We employ ∑ {xi,yi} log pθ(yi|xi, Zφ), which is unbiased estimate for −Hφ,θ(yi|xi, Z) based on Monte Carlo sampling. Given that the concrete information bottleneck Z is given, we expect that the Z allows us to generalize a novel task {xi, yi} by enhancing ∑ {xi,yi} log pθ(yi|xi, Zφ). However, in practice, Z induced by neural networks is more likely to memorize the training dataset including irreducible noises when the number of tasks is insufficient. The conventional methods accomplished maximizing ∑ {xi,yi} log pθ(yi|xi, Zφ) by simply finding the relation between yi and xi only in the training dataset. It means that the information in representation Z becomes increasingly identical to the information in xi in the meta training dataset. It does not satisfy the condition of information bottleneck for the context dataset D. To avoid this issue, we introduce I(Z, xi|D) as a regularization. By reducing dependencies of Z and xi, we can make the information bottleneck Z focus on D. It means that as p(Z|xi,D) and P (Z|D) get closer, the target input xi less influences Z, but D has a high correlation with Z. In other words, minimizing I(Z, xi|D) is to make Z and target xi more independent given context dataset D. It encourages latent variables Z = (wi, z) to pay more attention to context dataset D
I(Z, xi|D) = Exi [EZi [KL(q(Zi|xi,D)||q(Zi||D))]] = Exi [ Ewi [KL(q(wi|xi,D)||q(wi|D))] + Ez [KL(q(z|xi,D)||q(z|D))] ] (C.16) where, the q(wi|xi,D) follows the Weibull distribution and q(wi|D) follows the gamma distribution. In this study, q(wi|xi,D) can be modeled as the stochastic attention weights and q(wi|D) can be modeled as the key-based contextual prior qφ(wi|K). We can factorize all latent variables because the attention weight wi does not have a dependency on data points except for the specific data point {xi, yi}. We denote that wi has conditional independence over {wj}i 6=j given xi. For the global representation; z, to follow objective function of neural processes, we assume q(z|xi,D) = qφ(z|X,Y ) and qφ(z|D) = q(z|D). As mentioned in section 3, we suppose that the global representation z follows normal distribution. Therefore, we derive the regularization term I(Z, xt|D) have relation to KL Divergence terms in our loss function. From this fact, we recognize KL Divergence terms in our objective function helps latent variables give attention on context dataset.
To summarize the mutual information between yi and D, and the newly designed regularization, we found that
I(yi;D|xi) ≥ Iφ,θ(yi;Z|xi)− I(Z, xi|D) ≥ −Hφ,θ(y|xi, Z)− I(Z, xi|D) + Constant
(C.17)
Based on the graphical models in Figure 2 and assumptions of representations z and wi, we identify that this equation has relation to our objective function LTk
Iφ,θ(yi;Z|xi)− I(Z, xi|D) = ∑ {xi,yi} log pθ(yi|xi, Zφ)− Ez [KL(qφ(z|X,Y )|qφ(z|Xc, Yc))]
− EW [KL(qφ(W |X,D)|qφ(W |K(Xc)] + Constant = LTk + Constant
(C.18)
Finally, we can derive Theorem 1 as below
LTk(φ, θ) ≤ I(yi;D|xi)− I(Z, xi|D) (C.19)
From Equation C.19, the gradients of φ and θ with respect to LTk can be regarded as the direction of increasing I(yi;D|xi)−I(Z, xi|D), where, LTk is defined as the target likelihood and KL divergence of z and {wi}ni=1 in a single task.
LTk(φ, θ) = N∑ i=1 [ log pθ(yi|xi, z, ri)− KL ( qφ(wi|xi,D)|qφ(wi|XD) )] − KL ( qφ(z|X,Y )|qφ(z|D)
) (C.20) where XD is input features in the context dataset and qφ(wi|XD) follows key-based contextual prior as described in section 3.
D IMPLEMENT DETAILS
We referred to most of the architectures from the paper(Kim et al., 2019) and their released source code1. The information dropout and importance weighted ELBO were respectively borrowed from these papers(Kingma et al., 2015; Burda et al., 2016)23. The stochastic attention can be implemented based on Bayesian attention modules(Fan et al., 2020)4. We migrated and revised all codes to meet our purpose. In this chapter, we follow the notation of this paper (Lee et al., 2020).
D.1 (ATTENTIVE) NEURAL PROCESS
MLP Encoder We suppose that multi-layers modules(MLP) have the structure as Equation D.21, where l is the number of layers, din is the dimension of input features, dhidden is the dimension of hidden units and dout is the dimension of outputs.
MLP(l, din, dhidden, dout) = Linear(dhidden, dout) = ◦ (ReLU ◦ Linear(dhidden, dhidden) · · · )︸ ︷︷ ︸
l−2
◦ (ReLU ◦ Linear(din, dhidden))
(D.21)
The variants of neural processes have two types of MLP encoders: The deterministic path is used in the conditional neural process(Garnelo et al., 2018a), and the stochastic path is employed in the neural process(Garnelo et al., 2018b), attentive neural process(Kim et al., 2019) and ours. The deterministic path is to aggregate all hidden units by the MLP encoder.
h = 1 |c| ∑ j∈c MLP(lpre, dx + dy, dh, dh)([xj , yj ])
r = MLP(lpost, dh, dh)(h)
(D.22)
Instead, the stochastic path is to aggregate all hidden units and then feed-forward a single network to generate µφ1 and σφ1 . We obtain the stochastic path r via the reparameterization trick.
h = 1 |c| ∑ j∈c MLP(lpre, dx + dy, dh, dh)([xj , yj ])
µφ1 , σφ1 = MLP(lpost, dh, dh)(h) r = µφ1 + 2 ∗ σφ1
(D.23)
Attention encoder We introduce cross-attention to describe the dependency of context and target dataset. Let MHA be a multi-head attention (Vaswani et al., 2017) computed as follows :
Q′ = {Linear(dq, dh)(q)}q∈Q K ′ = {Linear(dk, dh)(k)}k∈K V ′ = {Linear(dv, dh)(v)}v∈V H = softmax(Q′K′/√dh)V ′
MHA(Q,K, V ) = LayerNorm(H)
(D.24)
Where, {dq, dk, dv} are respectively the dimensions of query, key and value components. The LayerNorm is the layer normalization in terms of heads.
MLP Decoder The architecture of MLP decoder is similar to the stochastic encoder. In case of CNP(Garnelo et al., 2018a) and NP(Garnelo et al., 2018b), this decoder transforms all representation z and the input feature {xi}i∈T to target distribution, the normal distribution, {µi, σi}i∈T .
1https://github.com/deepmind/neural-processes 2https://github.com/kefirski/variational_dropout 3https://github.com/JohanYe/IWAE-pytorch 4https://github.com/zhougroup/BAM
(µy, σy) = MLP(ldec, dx + dh, dh, 2dy)([z, ri, xi]) (D.25)
Meanwhile, in case of attentive neural process (Kim et al., 2019) and ours, the inputs of this decoder are the global representation z, local representation {ri}i∈T and the input feature {xi}i∈T .
(µy, σy) = MLP(ldec, dx + dh + dh, dh, 2dy)([z, ri, xi]) (D.26)
The detailed information of each architecture is described in Table D.1.
In this paper, we set the dimension of all latent variables as 128, namely dh = 128. The number of heads in multi-head attention is 8, which is the same as the original paper of attentive neural processes (Kim et al., 2019). For all models and all experiments, we use the Adam optimizer(Kingma & Ba, 2015) with the learning rate 0.001, and we set the number of update steps as 200000.
For training and evalation, we used AMD Ryzen 2950X(16-cores), RAM 64GB and RTX2080Ti. The 1D regressions has a batch size of 1 and a total of 160 batches. The Sim2Real configures the batch size to be 10 and the total number of batches to be 16. The MovieLenz-100k has a batch size of 1 and the number of batches to be 459. The image completion task consists of 227 batches and its size is 4.
D.2 MODEL ARCHITECTURE
We graphically show the architecture of the proposed method in Figure D.1. This model has two types of encoder as attentive neural process(Kim et al., 2019). The encoder parameters φ consists of φ1, φ2. The φ1 is responsible for the local representation ri and the φ2 is responsible for global representation z. The encoder of global representation is same as neural process(Garnelo et al., 2018b), however, the encoder of local representation is different from the standard cross attention(Vaswani et al., 2017). After obtaining standard attention weight wstandard, we introduce reparameterization trick for wi, which follows the Weiubll distribution(Fan et al., 2020). The important thing is that the key conceptual prior can be made by MLPφ13 ({xj}j∈C). The decoder is the same as the attentive neural process(Kim et al., 2019).
D.3 ALGORITHM
The proposed method requires hyper-parameters k and β for reparameterization of the Weibull distribution and KL divergence between the Weibull and the gamma distribution. We conduct the grid searches for k to find the best value. We identify that the proposed method with k = 300 adequately captures the dependency and generates noises to avoid memorization for all experiments. In the case of β, we follow the setting of the Bayesian attention module(Fan et al., 2020). We suggest the entire procedure of our algorithm as follows:
D.4 COMPARISON OF PROBABILISTIC GRAPHICAL MODELS FOR VARIANTS OF NPS
We describe NPs with probabilistic graphical models to differentiate the variants of NPs. The Neural process employs mean aggregate function and reparameterization trick to obtain a global context representation z(Garnelo et al., 2018b). This model follows Figure D.2a. The Attentive neural process expedites the multi-head cross attention to obtain local representation ri with global context representation z. We show that the graphical model for ANP is the middle of Figure D.2, all variables in the attention mechanism are regarded as the determinstic variable.
In this work, we design that all latent variables contain stochasticity and achieves by the reparameterization trick. By Bayesian attention module, we present our graphical model as shown in Figure 2
Algorithm 1: Neural Process with stochastic attention Input: Task distribution p(T ), the shape parameters of weibull distribution and gamma distribution : k = 300(Image completion task : k = 100), β = 1, Stepsize : γ output: encoder parameters : φ = {φ1, φ2}, decoder parameters : θ Initialization : φ and θ randomly; while not converged do
Sample tasks {Tk}Kk=1 from p(T ) for all Tk 3 {T } do
Sample context dataset (Xc, Yc) = {xj , yj}Mj=1 and target dataset (Xt, Yt) = {xi, yi}Ni=1 from the task Tk
Sample random noises 1 ∼ Unif(0, 1) and 2 ∼ N (0, 1) 1 Local representation {ri}Ni=1:
Q, K, V← fφ1(Xt, (Xc, Yc)) wstandard ← Softmax(QKT/ √ dk) (Reprameterization sampling of weibull distribution for W ) λ← wstandard ∗ Γ(1 + 1/k) W = {w1, · · · , wn} ← λ(− log(1− 1))1/k W = {w1, · · · , wn} ← { wi∑Wi , · · · , wn∑Wi } (Parameters of the prior distribution for W ) α← fφ1(K)
{ri}Ni=1 ← {wiV, · · · , wnV } 2 Global representation z :
µφ1 , σφ1 ← Aggregator(fφ2({xj , yj} M j=1)) (Reprameterization sampling of normal distribution for z) z = µφ2 + 2 ∗ µφ2
3 Decode p(Yt|Xt, z, {ri}Ni=1) : µy, σy ← (fθ(Xt, z, {ri}Ni=1)) 4 Evaluation loss : Lφ,θTk = ∑N i=1 [ log pθ(yi|µy,i, σy,i)− KL ( qφ2(z| {Xt, Yt})|qφ2(z| {Xc, Yc}) ) −
KL ( qφ1(wi|xi, Xc)|qφ1(wi|Xc) )] end Update φ← φ+ γ∇φ 1|T | ∑ Tk L Tk
Update θ ← θ + γ∇φ 1|T | ∑ Tk L Tk
end
E 1D REGRESSIONS
For the synthetic 1D regression experiment, we set dx = 1, dy = 1. The number of layers in encoder and decoder in all baselines is respectively lenc = {lpre = 3, lpost = 1} = 4 and ldec = 3. We set the dimension of latent variable as dh = 128.
When it comes to data generation process, the training data is generated from the Gaussian process with RBF kernel. For each task, we randomly generate x ∼ Unif(−2, 2) and then y is the function value by the Gaussian process with RBF kernel, k(x, x′) = s2 · exp(−‖x−x′‖2/2l2). To validate our model, we establish two types of datasets in the this 1D regression experiment.
First, we set the parameters of the RBF kernel as s = 3 and l = 3. Hence, as shown in Figure E.3a, all datasets are drawn from continuous and smooth functions. Second, we consider the noisy situations in the first scheme. We intend that the suggested scheme represents the actual real-world situation. Unlike the existing studies (Garnelo et al., 2018a; Kim et al., 2019; Lee et al., 2020), we modify the RBF kernel function adding a high frequency periodic kernel function k(y, y′) = s2 · exp(−2 sin2(π‖x− x′‖2/p)l2). Lee et al. (2020) proposed a noisy situation by using random noises sampled by t-distribution; however, these noises often have exaggerated values so that the generated function does not have any tendency and seems to be entire noises. On the other hand, the function generated by Gaussian processes with a high frequent periodic kernel is smooth but is satisfied with a random function every trial. Thus, this function does not interfere with the smoothness of the RBF GP function and maintains the smoothness of all support ranges. Therefore, we decide to use the dataset generated by the RBF GP function with periodic noises to synthetically test all baselines and the proposed method for the robustness of noises. The sampled dataset is graphically presented in Figure E.3b.
(a) The RBF kernel GP (b) The RBF kernel GP with peri-odic noises
Figure E.3: Comaprison of RBF GP functions in 1D regression problems
To generate function values of the Gaussian process, the GPy library5 provides various functions compatible for PyTorch6. In the GPy, ‖x − x′‖2 can be regarded as the pre-defined vector {1, . . . , freq}T {1, . . . , freq}. We set the parameters of the periodic kernel as freq = 30, p = 2π, s = 1. The generated functions for training datasets are shown in Figure E.3.
To test generalization to a novel task, we introduce other functions such as the Matern32 kernel GP and the periodic kernel GP as shown in Lee et al. (2020). In this experiment, we expect the trained models to capture context points despite different functions. Naturally, all models perform well on the RBF GP function because the meta-training set is generated from RBF GP functions; instead, we test all baselines on other types of functions such as the Matern32 kernel GP and the Periodic kernel GP as well as the RBF kernel GP. For the Matern32 kernel k(x, x′) = s2(1+ √ 3‖x−x′‖) exp(− √ 3‖x−x′‖) and the periodic kernel is same as mentioned earlier. The shared parameters of these three GPs are same as s = 3, l = 3. The periodic kernel requires another parameter freq = 10, which is the default value in the GPy.
To elaborate the meta-training framework, the minimum number of context points is 3, and the minimum number of target points is 6, including the context data points. The maximum number of context points and target points is respectively 97 and 100. In the following subsection, we report experimental results, including clean and noisy situations.
5https://github.com/SheffieldML/GPy 6https://pytorch.org
Table E.4: Results of likelihood values on the synthetic 1D regression without noises. Bold entries indicates the best results.
RBF kernel GP Matern kernel GP Periodic kernel GP context target context target context target
CNP 1.136±0.044 0.762±0.090 -0.465±0.106 -3.684±0.222 -2.354±0.218 -9.612±0.0287 NP 0.687±0.030 0.312±0.127 -1.222±0.055 -3.546±0.268 -2.252±0.115 -6.038±0.195 ANP 1.326±0.004 0.819±0.023 0.963±0.039 -1.538±0.083 -0.193±0.219 -7.447±0.278
(Weight decay; λ = 1.0e− 3) CNP 1.167±0.025 0.827±0.051 -0.310±0.067 -3.135±0.178 -2.278±0.170 -8.840±0.259 NP 0.695±0.031 0.297±0.051 -1.297±0.201 -3.583±0.536 -2.577±0.223 -6.489±0.268 ANP 1.355±0.002 0.924±0.035 1.077±0.023 -1.753±0.089 0.379±0.080 -7.663±0.325
(Importance Weighted ELBO; k = 5) CNP 1.170±0.023 0.803±0.083 -0.443±0.130 -3.704±0.296 -2.422±0.151 -9.639±0.330 NP 0.728±0.028 0.255±0.156 -1.476±0.054 -4.208±0.175 -3.042±0.399 -7.463±0.624 ANP 1.344±0.004 0.889±0.037 0.854±0.050 -1.854±0.114 -0.483±0.198 -7.900±0.201
(Bayesian Aggregation) CNP 1.282±0.014 0.978±0.019 -0.141±0.140 -3.443±0.294 -2.016±0.168 -9.703±0.221 NP 0.379±0.023 0.069±0.051 -0.751±0.074 -1.811±0.0.066 -1.465±0.092 -3.308±0.177 (Functional representation) ConvCNP 1.351±0.002 0.891±0.024 1.336±0.008 -1.941±0.066 1.020±0.062 -9.634±0.249 ConvNP 1.318±0.012 0.905±0.029 1.287±0.015 -1.806±0.169 0.953±0.057 -9.580±0.306
(Regularization for local representation) ANP (dropout) 1.348±0.001 0.866±0.029 0.902±0.051 -1.753±0.084 -0.520±0.211 -8.052± 0.324 Bootstrapping ANP 1.347±0.005 0.895±0.026 0.790±0.044 -1.834±0.084 -0.983±0.309 -8.463± 0.343 Proposed Method 1.343±0.006 0.937±0.040 1.170±0.013 -1.708±0.043 0.681±0.052 -7.807±0.399
E.1 EXPERIMENT RESULT ON RBF GP FUNCTIONS
We train all baselines on the RBF GP functions without noises. We demonstrate experimental results in terms of predictability and context set encoding.
As shown in Table E.4, all models are capable of fitting the RBF GP function; meanwhile, all models are degraded in cases of Matern kernel GP and Periodic GP. Unlike all baseline models, of which performances drop substantially, the proposed method has relatively small degradation. Particularly, the proposed method records that the likelihood value of context points at Matern kernel GP and Periodic GP is respectively 1.170 and 0.680. Meanwhile, the best performance among baselines is 1.077 and 0.379 by ANP with weight decay. This result shows that the proposed method performs better than all baselines. The graphical results are described in Figure E.4.
E.2 EXPERIMENT RESULT ON RBF GP FUNCTIONS WITH NOISES
As mentioned early in the current section, we train all models on the RBF GP functions adding the random periodic noise. Looking at the column of RBF kernel GP in Table 1, most of the baselines are degraded due to noisy situations. However, the performances of all baselines are improved in Matern GP and Periodic GP. We guess that this phenomenon can be explained as mitigating memorization issues by injecting random noise to the target value (Rajendran et al., 2020). We present how effectively random noises improve generalization performances for all baselines and our method in this experiment. However, we can recognize that injecting random noises cannot be a fundamental solution. As seen in Appendix G, we emphasize that capturing and appropriately utilizing context information when predicting target values is a fundamental solution to avoid memorization in a given situation. The experimental result is described in Table 1 on manuscript and the detailed graphical explanation is suggested in Figure E.5.
Figure E.4: 1D regression results by NPs trained on the RBF kernel function without periodic noises. All baselines and ours fit the RBF GP function, but degrade their performances in cases of Matern and Periodic kernel GP.
Figure E.5: 1D regression results by NPs trained on the RBF kernel function with periodic noises. All models improve performance in Matern and Periodic GP than Figure E.4. The proposed method outperforms baselines and regularization methods. Even this model achieves better performance than the result on RBF GP without noises.

heading: F PREDATOR PREY MODEL

text:
We follow the experimental detail in this paper(Lee et al., 2020; Gordon et al., 2019). This experiment is designed to evaluate all baselines capable of adapting new tasks, which have slightly different task distributions. It is called "Sim2Real". We assume that, in this situation, we easily obtain simulation data, but it requires a high expense to collect real-world data points. We try to train all baselines on the dataset generated by simulations and test them on real data sets that are relatively small compared to the simulation data.
The way of simulated data is generated from the Lotka-Volterra model(LV model)(Wilkinson, 2018). Note thatX is the number of predators, and Y is the number of prey at any time step in our simulation. According to the explanation, one of the following four events should occur. The following events require the parameter θ1 = 0.01, θ2 = 0.5, θ3 = 1, θ4 = 0.01
1. A single predator is born according to rate θ1 ·X · Y , increasing X by one. 2. A single predator dies according to rate θ2 ·X , decreasing X by one. 3. A single prey is born according to rate θ3 · Y , decreasing Y by one. 4. A single prey dies(or is eaten) according to rate θ4 ·X · Y , decreasing Y by one.
The initial X and Y are randomly chosen. On the other hand, Hudson’s bay lynx-hare dataset follows a similar tendency to the LV model; however, it is oscillating time series because it has been collected in real-world records. There were unexpected outliers, unexplained events, and noise. For the detailed explanation, refer Gordon et al. (2019)’s work. All simulation codes are available on this URL7. The generated simulation data can be graphically shown as the left side of Figure F.6.
In this experiment, we set dx = 1, dy = 2, and all remaining settings are the same as the 1D regression experiment except that the number of context points is at least 15 and the extra target points is also at least 15. When training models, we use the dataset generated by the LV model and test on the lynx-hare dataset. The experimental result is shown in Table 2, and the prediction performance can be graphically shown in Figure F.6.
We conduct additional experiments to validate whether the random periodic noise positively influences the performance. As conducted in the 1D regression experiment, we utilize the periodic kernel GP function, which has very high frequency freq = 100 to add noises to the training dataset. As shown in Table F.5 compared to Table 2, We recognize that adding random periodic noise empirically improves all baselines and our method. Among models, our models perform best due to the full representation of context and target datasets. The prediction results can be graphically shown in Figure F.6b
Table F.5: Predator-prey model results. All models are trained on data with periodic noises.
Simulation Real data context target context target
CNP -0.181 -0.308 -2.420 -2.789 NP -0.641 -0.739 -2.464 -2.710 ANP 0.645 0.290 -0.634 -1.962
(Importance weighted ELBO (s = 5)) CNP -0.284 -0.390 -2.484 -2.864 NP -0.527 -0.628 -2.335 -2.641
(Bayesian Aggregation) CNP -0.060 -0.171 -2.393 -2.776 NP -0.357 -0.458 -2.260 -2.618
(Functional representation) ConvCNP 2.395 1.679 1.879 -0.205 ConvNP 2.368 1.687 1.691 -0.521
(Regularization for local representation) ANP (information dropout) -0.749 -0.990 -1.766 -2.458 Bootstrapping ANP 0.326 -0.008 -1.183 -2.008 Ours 2.745 1.819 2.699 -0.076
7https://github.com/juho-lee/bnp
(a) On the clean data (b) On the noisy dataset.
Figure F.6: Prediction results on the hare-lynx data. (left) All models are trained on the clean dataset. (right) All models are trained on the noisy dataset.

heading: G ANLAYSIS OF CONTEXT EMBEDDINGS VIA ATTENTION WEIGHTS IN 1D REGRESSIONS AND THE LYNX-HARE DATASET

text:
The attention mechanism can explicitly display the distance between context and target data points via attention weights. Especially, when the dimension of features is one, x ∈ R1, the distance between features is simply calculated and sorted, so allowing for straightforward comprehension when the attention weights are shown in heat-maps. As a result, we present several heatmaps to compare ours with baselines in both clean and noisy 1D regression datasets. The horizontal axis in these graphs represents the value of features in the context dataset, while the vertical axis represents the value of features in the target dataset. The best pattern for this heat-map is diagonal because all feature values are arranged in ascending order. Additionally, we will provide the simplified heat-map that take into account only target points with same value in the context dataset. Due to the fact that the original version has one-hundred target points, labels and ticks are necessarily small. We guess that readers may be unable to decipher the detailed information such as labels and attention scores. This simplified version allows the readers to instantly grasp what we attempt to convey by exhibiting a more distinct diagonal pattern. From Figure G.7 to Figure G.9, on the left side, we show a simplified version of the attention score; on the right side, we show the result for the entire target dataset. Plus, we provide Sim2Real’s attention heat-maps in both clean and noisy environments. As a result, our discovery appears to be consistent.
First, the heatmaps of the 1D regression problem without perioidc noises are shown in Figure G.7. In this graph, all models including ours can accurately depict the similarity between the context and targete dataset. Although the attention scores for each model vary, it is clear that the majority pattern of all models is diagonal. Therefore, all models are trained using the clean dataset in the intended manner. However, other phenomena are found in the noisy data. We identify that the attentive neural process Kim et al. (2019) fails to capture contextual embeddings because the attention weights of all target points highlight on the lowest value or the maximum value in the context dataset. In the case of the Bootstrapping ANP Lee et al. (2020) and ANP with information dropout, the quality of heat-map is slightly improved, but it still falls short of ours. This indicates that the present NPs are unable to properly exploit context embeddings because the noisy situations impair the learning of the context embeddings during meta-training. On the other hands, even in the noisy situation, the attention score in ours still appear clearly. Second, we include heatmaps for the Sim2Real problem. As seen in Figure F.6, the context and target datapoints are much too many for the label information and ticks to be recognized. Hence, we recommend that readers verify the presence of diagonal patterns rather than examining detailed numerical values. As seen in Figure G.9a, all models are capable of capturing properly similarity between the context and target datasets. However, as with 1D regression, the diagonal pattern of the baselines is disrupted as seen in the left graph of Figure G.9b; nevertheless, ours retains the ideal pattern in both clean and noisy situations.
When comparing our model’s heat-map pattern in clean and noisy environments, there is a noteworthy point. Our model is capable of learning adaptively how to focus on certain context datapoints depending on the extent of dirty data in the meta-train dataset. The model is trained on the clean dataset to take into consideration nearby points as well as the corresponding point in the context dataset, hence, the heat-map gradually changes. This is because the clean dataset has smooth values, yi,j , near a certain feature xi,j . Meanwhile, in the noisy dataset, the model is trained to focus exclusively on corresponding points to the context dataset. Hence, the heat-map in Fig E.5 (b) indicates that attention score of target datapoints that includes the context dataset has a high value, whilst the remainder points treat all context datapoints uniformly. This phenomena occurs because there is less correlation between adjacent features xi,j and its labels yi,j in the noisy situation during the meta-training.
Figure G.7: Heatmaps of asscociated attention weights in 1D regression problems when test datasets are sampled from RBF and Periodic kernel functions. All models are trained on the clean dataset. (Left) A simplified heat-map that takes into account only target datapoints with same value in the context dataset. (Right) A heat-map that takes into account entire target datapoints.
Figure G.8: Heatmaps of asscociated attention weights in 1D regression problems when test datasets are sampled from RBF and Periodic kernel functions. All models are trained on the data with periodic noises. (Left) A simplified heat-map that takes into account only target datapoints with same value in the context dataset. (Right) A heat-map that takes into account entire target datapoints.
(a) (b)
Figure G.9: Heatmaps of asscociated attention weights in the hare-lynx dataset. (Left) A simplified heat-map that takes into account only target datapoints with same value in the context dataset. (Right) A heat-map that takes into account entire target datapoints. (a) All models are trained on the clean dataset by the lotka-volterra simulation (b) All models are trained on the lotka-volterra simulation with periodic noises.

heading: H ABLATION STUDY : RELATIONSHIP BETWEEN REGULARIZATION OF PAYING MORE ATTENTION TO THE CONTEXT DATASET AND CONTEXT EMBEDDINGS

text:
This section discusses the effect of suggested regularization, paying more attention to the context dataset on attention scores. We demonstrated how the proposed regularization term results in an increase in learning stability and embedding quality. Prior to conducting a thorough comparison of stochastic attention with and without regularization, we analyze learning curves between baselines, such as CNP, ConvCNP and ANPs in 1D regression under a noisy situation. Because the proposed regularization naturally incorporates the attention mechanism, we begin by comparing the proposed model to NPs that do not employ the attention mechanism. We choose CNP with Bayesian Aggregation, NP with Bayesian Aggregation(Volpp et al., 2021) and ConvCNP(Gordon et al., 2019) as baselines, taking into account prediction performance as demonstraed in Table 1. As shown in Figure H.10a, the negative loglikelihood of CNP_BA and NP_BA have converged to about −10.0 and 0.0 However, the ConvCNP and ours record values around −30.0. In the case on ConvCNP, this method optimally computes similarity between context and augmented data points (called grid) that include target points with a high probability using the thereotically well-defined functional representation by RBF kernel function. On the other hand, the proposed regularization approaches the ideal loglikelihood value without the assistance of extra data points. The proposed can train how to establish similarity between context and target points and predict labels of target points during meta-training. Second, we compare training curves between ANP, ANP(dropout)(Kim et al., 2019), Bootstrapping ANP(Lee et al., 2020) and ours. This comparison indicates that while all methods make use of the attention mechanism, the chosen baselines exclude stochastic attention and regularization; I(Z, xi|D). When we look at Figure H.10b, we observe the same results as shown the quality of context embeddings in Appendix G. While the Bootstrapping ANP had superior training curves to ANP and ANP(dropout), but it was unable to be match ours.
(a) (b)
Figure H.10: Comparison of training curves with their respective baselines. (a) The baselines that do not employ the attention mechanism. (b) The baselines that employ the attention mechanism.
We examine the general effect of the proposed regularization to the stochastic attention by altering hyper-parameter K. We conduct experiments in which all models use same generative process for stochastic attentions, as desribed in Figure 2, but with varied hyper parameter, K ∈ {1, 10, 20, 30, 40, 50, 60, 70, 80, 100, 300, 500} and the presence of KL[qφ(wi|xi, Xc)|qφ(wi|Xc)]. To begin, when we analyze Table H.6, we observe that variation of the hyper-parameter K has little influence on the prediction performance of 1D regressions. If the models properly converge, the prediction outcomes should be statistically indistinguishable. However, the proposed regularization has an effect on model’s learnability. While all experiments cases that employ the proposed regularization converge, a few models with the hyper-parameter K ∈ {10, 50, 70} that do not use the proposed regularization, do not. Additionally, we demonstrate this fact via negative log likelihood values in the meta-training, as seen in Figure H.11. As Table H.6 indicates, all models that use the proposed regularization have converged, whereas some models that do not use the proposed regularization and use the hyper-parameter K ∈ {10, 50, 70}, have significantly different training
curves when compared of converged models. Therefore, we anticipate that the proposed method will ensure learnability and convergence regardless of a hyper-parameter value.
Table H.6: Prediction performance of ours varying the hyper-parameter K and regularization. All models employ the generative process of stocahstic attentions. The hyphen means a model converges.
No Regularization Regularization RBF Periodic RBF Periodic
Models Trainable context target context target Trainable context target context target
K=1 - 1.383±0.000 0.421±0.008 1.382±0.000 -0.322±0.028 - 1.376±0.001 0.405±0.007 1.376±0.001 -0.347±0.030 K=10 7 0.543±0.007 0.199±0.009 0.301±0.015 -0.645±0.030 - 1.383±0.000 0.417±0.008 1.369±0.002 -0.402±0.025 K=20 - 1.380±0.001 0.431±0.008 1.380±0.000 -0.355±0.029 - 1.377±0.003 0.408±0.011 1.381±0.001 -0.366±0.017 K=30 - 1.381±0.001 0.356±0.011 1.380±0.000 -0.370±0.027 - 1.382±0.000 0.452±0.007 1.381±0.000 -0.423±0.026 K=40 - 1.380±0.001 0.395±0.009 1.381±0.000 -0.313±0.031 - 1.379±0.002 0.446±0.008 1.380±0.001 -0.399±0.021 K=50 7 0.730±0.012 0.267±0.022 0.520±0.012 -0.642±0.043 - 1.379±0.002 0.353±0.004 1.380±0.001 -0.501±0.028 K=60 - 1.378±0.003 0.433±0.007 1.381±0.001 -0.460±0.029 - 1.380±0.001 0.349±0.004 1.380±0.001 -0.338±0.025 K=70 7 0.642±0.005 0.235±0.010 0.401±0.017 -0.644±0.029 - 1.373±0.001 0.429±0.008 1.377±0.000 -0.370±0.024 K=80 - 1.379±0.002 0.423±0.011 1.381±0.001 -0.374±0.034 - 1.375±0.002 0.443±0.009 1.379±0.002 -0.353±0.030 K=100 - 1.375±0.002 0.373±0.007 1.378±0.001 -0.436±0.030 - 1.377±0.001 0.457±0.009 1.378±0.001 -0.391±0.034 K=300 - 1.380±0.002 0.393±0.006 1.382±0.000 -0.369±0.026 - 1.379±0.002 0.417±0.006 1.382±0.001 -0.346±0.024 K=500 - 1.383±0.000 0.359±0.008 1.383±0.000 -0.316±0.030 - 1.381±0.001 0.402±0.009 1.380±0.001 -0.352±0.032
(a) No regularization. (b) Regularization
Figure H.11: Comparison of training curves varying the hyper parameter K. All models employ the generative process of stocahstic attentions.
We study the learned attention scores in detail to verify the proposed regularization’s efficiency. To conduct fair comparison, we fix the number of context points and choose target points that have same value as the context dataset. The ideal scenario is that all scores in the diagonal components have identical values and little variation regardless of the placement of target points as the attention mechanism should focus exclusively on context and target dataset.
As seen in the case of no regularization in Figure H.12, despite somewhat varied scores, all converged models exhibit the same diagonal pattern. As a result, these models present similar prediction performance. We argue that 1D regressions do not need precise context embeddings. However, when we compare the attention scores with and without regularization, we discover findings regarding the quality of context embeddings. In the absence of regularization, the values of attention weights changes according to the hyper-parameter K. Certain models place a less emphasis on the diagonal components than others, while others excessively place a greater emphasis on them. Without the regularization, the attention scores of diagonal components range from 0.3 to 0.64. We instantly notice that without regularization, the attention heat-map of stochastic attention has an inconsistent degree of brightness in relation to the hyper parameter K. In particular, the heatmaps of K = 60, 100 is too bright in comparison to other heat-maps, whilst the heatmaps of K = 40 is too dark. While the diagoanl pattern is presented, there is no consistency in the brightness of any heat-maps.
On the other hand, the attention scores of regularized models exhibit little fluctuation. Regardless of the hyper parameter K, these scores are closed to 0.5. Table H.7 contains the quantitative result. In the regularized model, the variance of attention scores for diagonal components is 0.003 and the mean value of attention scores is 0.469. Nonetheless, the mean of the model without the regularization is 0.367 and the variance is 0.026. This demonstrate that regularized models consistently
Diagonal Off-Diagonal Mean Variance Mean Variance
No Regularization 0.367 0.026 0.059 0.001 Regularization 0.469 0.003 0.070 0.002
Table H.7: Descriptive statistics of attention scores between diagonal and off-diagonal compoenents in cases of regularization, I(Z, xi|D) and no-regularization
properly emphasize on the context dataset according to property of context and target dataset. The regularization cases in Figure H.12 illustrate this fact graphically. All heatmaps of stochastic attention that employ the proposed method regularization have very a comparable brightness. We identify that the proposed regularization ensures learning stability by ensuring the quality of context embeddings remains consistent indepedent of the hyper-parameter K. As a consequence, we declare that the proposed method increases model’s insensitivity to hyper-parameters K by preserving the context embeddings consistently.
Figure H.12: Comparison of attention scores varying the hyper-parameter, k and the presense of regularization, in 1D regressions drawn from the RBF kernel GP.
The intriguing fact is that the difference in attention scores for the diagonal component is rather significant within the same models. Since the values of target points are same as the context dataset, the attention scores should be constant. For instance, when K = 40 and no regularization is used, the attention scores are as {0.64, 0.64, 0.54, 0.47, 0.64, 0.54, 0.57, 0.62, 0.64, 0.64}. The attention scores for the same model with regularization are {0.51, 0.52, 0.44, 0.43, 0.45, 0.43, 0.45, 0.46, 0.5, 0.52}. The variance in absence of regularization is 0.003, whereas the variance in the presence of regulariza-
(a) No regularization. (b) Regularization
Figure H.13: Mean and variance for attention scores of diagonal components varying the hypereparameter K and regularization.
(a) No regularization. (b) Regularization
Figure H.14: Mean and variance for attention scores of off diagonal components varying the hypereparameter K and regularization.
tion is 0.001. As a result of the consistent scores across all target points ranges, we realize that the regularized model improves context embeddings.
The detailed analysis is shown in Figure H.14. While the mean values of attention scores vary between individuals, the proposed regularization assures that attention scores are constantly near 0.5 value. From the perspective of NP’s aim, the proposed regularization enbales models to achieve those objectives. The attention scores of all target points must not be influenced by feature magnitudes and their regions. Consequently, this method has to concentrate only on the similarity between features. The regularization aims to derive an appropriate attention mechanism that evenly weights across important regions and eliminates biases during the meta-training. It leads to achieving task adaptation, as it equally prioritizes context data points regardless of features magnitude. Whenever the novel arises, the regularization ensures that the models retain the quality of their context embeddings.
We investigate more research into the cases of not converged models. As seen in Figure H.15, there are completely distinct phenomena from what we have observed so far. When models are not converged, all attention scores remains constently uniform across all region. As a result, the prediction performance degrades. However, when the models use the proposed regularization, the attention scores are comparable to the others. This means that the regularization is crucial for preventing models from trapped in the local optima. Additionally, as seen in Figure H.11, this regularization enables models can approach to maximize log likelihood values. By examining these phenomenon, it is clear that the approach proposed in this study performed as a regularization role.
Figure H.15: Comparison of attention scores when the models without regularization are not properly trained, K ∈ {10, 50, 70}, in 1D regressions drawn from the RBF kernel GP. (left) All methods do not utilize the suggested regularization. They employ the reparameterization trick exclusively for attention scores. (right) All methods employ the suggested regularization.

heading: I MOVIELENZ-100K DATA

text:
We follow the experimental detail in the paper (Galashov et al., 2019). The way to split into train, test, validation dataset is described in this work8. Briefly, all ratings are first divided into two datasets, which respectively have user IDs and are not duplicated each other. We define that one is the meta-training dataset and the other is the meta-test dataset. In this experiment, we evaluate baselines that generalize to a new customer with small history data points. To explain a training regime, the number of context is at least three and the objective function is computed with the remaining data points. All users do not have the same amount of ratings so that the number of data points is varying during meta-training. When testing baselines, we sample a few ratings from unobserved users and then evaluate the likelihood values for the remaining points. To exclude users’ unique information, we drop the User ID from the set of features and only use general features such as age, occupation. Lastly, in this experiment, we set dx = 44, dy = 1, and all remaining settings are the same as the 1D regression experiment. The detailed explanation is in Galashov et al. (2019)’s work. The likelihood values on the MovieLens-10k is reported in Table I.8.
To compare RMSE scores with existing methods for Movielens-100k data, we suggest evaluating NPs using u1.base and u1.test. As mentioned, the prediction by NPs is based on given context data points. First, we randomly choose a customer in the u1.test. Second, we randomly sample a small number of ratings from u1.base, which is used as the meta-training data. In this experiment, we decided to draw ten samples from u1.base and employ these points as the context data set. The ten ratings is relatively a short history compared to the size of the u1.base. Third, we perform predictions for the data points in the u1.test based on the context data points. The important thing is that context and target data points are sampled from the same user but the different data set. Plus, we do not touch in information in the test set, at least. Based on the relationship between sampled context points and target points, the performance of NPs is somewhat variable, so we report five runs and average these results. The experimental result is in Table 3.
8https://tinyurl.com/yyfzlg2x
J IMAGE COMPLETION
In the image completion task, we follow the training regime suggested by NPs(Garnelo et al., 2018a;b). In this regime, dx means the pixel location and dy means the color RGB data. Namely, we set dx = 2 and dy = 3. For high capacity representation, the dimension of all hidden units is determined as dh = 512. All remaining settings, including layer sizes, are the same as the 1D regression experiments.
When it comes to image completion tasks, according to these papers, NPs are known to complete facial images given partial pixels. Although a few pixels are provided, NPs are capable of understanding the common appearance of a human face. These models generate eye, mouth, and nose even if only a small number of local information is given, which is generally hard to recognize. The Conditional neural process properly performs; however, it often loses its context representations. The Attentive neural process tackles this issue to employ an attention mechanism. Instead of the encoders consisting of MLP layers, the self-attention can improve the quality of pixels generated at the context data points. However, as mentioned on subsection 4.2, scores of generated pixels in unseen regions are still problems. Even though many pixels are provided, the quality of completed images is not improved. In this work, the proposed method concentrates on given contextual pixels and complete images based on that information. Hence, the proposed method completes an image with high uncertainty when few pixels are given. Plus, it can be seen that the higher the pixels information, the higher the quality of generated images. The numerical result is presented in Table 2, the graphical evidences are shown in from Figure J.16 to Figure J.19.
The proposed method allows for completion of images using only partially provided pixels, so it enables to complete larget images than the image size in the training dataset. We produce images of various sizes using the proposed method, which is trained on the CelebA dataset with size of 32 × 32. The completed images are presented in Figure J.20. When we qualitatively evaluate the completion results, the proposed method properly perform for generating large images without sacrificing performance. All completed images are quite close to the ground truth. As a result, we conclude that the proposed method can be used for task adaptation since it can complete images that the model have never encountered during meta-training and whose sizes differ from the training dataset. In particular, while the percentage of contributed pixels to the total number of pixels remains constant, the quality of completed images are enhanced according to the amount of provided pixels. We understand that the number of pixels is more significant than the ratio of context points. This is because the model has a sufficient pixel counts to work with when images are large. For instance, employing only 300 pixels as a context dataset is insufficient to complete the images. Nonetheless, the completed images with 300 pixels are comparable to those with higher pixel counts and is of higher quality than the baselines in Figure J.18.
There is an interesting discovery of uncertainty values in the image completion task. Basically, we expect that many pixels reduce the uncertainty of generated images. However, the existing methods stably generate facial images, but uncertainty values do not decrease even if available pixels increase. The proposed method makes high variance for a completed image under the small number of context points. As shown in Figure J.21, it can decrease the uncertainty of completed images when the number of context points is large. From this fact, we identify that the proposed method performs in an intended manner.
Figure J.16: Completed images using 50 pixels
Figure J.17: Completed images using 100 pixels
Figure J.18: Completed images using 300 pixels
Figure J.19: Completed images using 500 pixels
(a) Example 1
(b) Example 2
(c) Example 3
Figure J.20: Our completed images in a variety of sizes. The training dataset consists of the CelebA images with a size of 32× 32. The completed images range in size from 32× 32 to 80× 80. Each image is completed with 0.3 of the overall pixels. As the image size increases, the following values are used as the number of context points;{300, 690, 1230, 1920}. Both the context dataset and its completed image are displayed. Additionally the most right column has the ground truth image. Each image is shown in proportion to its image size.
Figure J.21: Comparison of quality of uncertainty by CNP with Bayesian aggregation and Ours.