Sequential Neural Processes (SNP) is a new class of models that temporally extends Neural Processes (NP) and can meta-learn a sequence of stochastic processes. This learned function however under-fits the provided contexts as is also the case in NP. Applying attention to the contexts resolves this in NP but simply extending this to SNP and applying attention on a buffer of context history is sub-optimal as our findings show. In this paper, we propose Attentive Sequential Neural Processes (ASNP) which resolves the under-fitting in SNP by introducing a novel imaginary context, modeled as a latent variable, over which the attention can then be applied. We evaluate our model on 1D Gaussian Processes regression and 2D moving MNIST/CelebA regression. We apply ASNP to implement Attentive Temporal-GQN and further evaluate on the moving CelebA task.
This paper combines ideas from attentive and sequential neural processes to incorporate an attention mechanism to the existing sequential neural process, which results in an attentive sequential neural processes framework.
While the idea is somewhat interesting, I think this paper is technically vague and not well-motivated, which makes it hard for me to feel convinced that the problem exists and is non-trivial, and that the proposed solution is significant. Let me elaborate on my thoughts below:
First, the authors stated that SNP is subject to the underfitting problem that plagues NP but it is not clear to me why, in the temporal context of SNP, do we need to focus our attention on past contexts, which are no longer relevant. Could the authors please motivate this with a concrete application scenario? Without a concrete scenario, I do not feel very convinced that the problem exists.
Second, the argument that augmenting SNP with an attention mechanism is not trivial is somewhat contrived. In particular, the reason for this non-triviality is that (in the authors' own words) SNP assumes that it cannot store the past context as is -- so what if we simply store the past context & condition the representation on the entire history of past context instead? 
Apparently, this can come across trivially by replacing C_t with both C_<t and C_t in Eq. (2). This is in fact very similar to what the authors did in Eq. (4) which summarizes the generative process of ASNP -- the only difference is the generation of imaginary contexts, whose necessity is again questionable, as I elaborate next.
Third, the motivation for imaginary context is pulled from a very distant literature on how a human brain memorizes past experiences in a lossy memory consolidation, which only retains the most important sketches. In the context of ASNP, it is not, however, clear to me why this mechanism is necessary given that entire lossless memory can be stored except that without a lot of contexts, there is not a need for an attention component (as implied in first paragraph of Section 3) which is a contrived motivation.
Fourth, the technical exposition of this paper is too vague. Given that the key contribution here is about an attention component, the background review on ANP is surprisingly informal with no technical detail at all. For the other parts, the technical part is also mostly abstracted away -- what is presented is therefore not that much different from a typical generative model with latent variables, which makes it unclear whether there is a technical challenge here. 
In fact, from what I see, going from Eq. (2) to Eq. (4) is not much of a conceptual challenge and the execution of Eq. (4) (particularly the attention component described in Section 3.2) seems like a bunch of arbitrary engineering ideas which were put together to substantiate Eq. (4). 
Is there a technical challenge in the entire pipeline that should have been highlighted?
For the experiment, could the author compare the performance between ASNP and ASNP without the imaginery component (but with the attention mechanism)? It would be a good experiment to see if the imaginery component is necessary.
To summarize, I believe the paper in its current state is not well-motivated and appears very incremental given the prior works of SNP and ANP. Even its imaginery component, which is the key contribution here,  is, if I understand Eq. (3) correctly, not much different from context sampling of a NP.