The inference of Gaussian Processes concerns the distribution of the underlying function given observed data points. GP inference based on local ranges of data points is able to capture fine-scale correlations and allow fine-grained decomposition of the computation. Following this direction, we propose a new inference model that considers the correlations and observations of the K nearest neighbors for the inference at a data point. Compared with previous works, we also eliminate the data ordering prerequisite to simplify the inference process. Additionally, the inference task is decomposed to small subtasks with several technique innovations, making our model well suits the stochastic optimization. Since the decomposed small subtasks have the same structure, we further speed up the inference procedure with amortized inference. Our model runs efficiently and achieves good performances on several benchmark tasks.
PAPER SUMMARY:
This paper proposes a fast inference method for Gaussian processes (GPs) that imposes a sparse decomposition on the VI approximation of the posterior GP (for computational efficiency) using the KNN set of each data point. This is further coupled with armortized inference for better scalability. 
NOVELTY & SIGNIFICANCE:
This paper adopts a different approach of characterizing the VI approximation of a GP posterior than original VI approximation that was developed in Titsias (2009): Instead of characterizing the surrogate q(f_I) of p(f_I | Y) for a small collection of inducing inputs, the proposed method characterize q(f) directly where q(f) = int_f_I q(f_I) p(f | f_I)df_I.
This is, however, a somewhat strange direction which, to me, seems to raise extra issues that could have been avoided if one follows the conventional VI approximation:
(1) As the posterior surrogate is now directly over f instead of f_I, the number of variational parameters is now proportional to the data size which requires several (redundant) extra approximations including armortized inference & the lower-bound on the entropy term that admits a sparse decomposition.
(2) This at least creates the armortized and entropy gaps that decrease the expressiveness of the original VI. While I understand that this is in exchange for the ability to encode local information (via KNN) within the surrogate posterior, it is not clear to me why do we need to incur all these computational issues to incorporate such local information.
(3) For example, instead of forcing such local information in the posterior surrogate q(f), we could alternatively let it be reflected in the test conditional p(f_* | f_I, Y_n(*)) such that the test output depends on both the inducing output and a local partition of data (e.g., via K-mean), which has been previously explored in [*] and later incorporated in the conventional VI paradigm of Titsias (2009) without incurring extra intractability [**].
(4) This maintains the dense correlation between data points within the same neighborhood while allowing the VI surrogate to be more concisely specified and independent of the no. of training data points. Furthermore, it also helps avoid the data-bound overhead of computing a KNN per test point. 
[*] Local and global sparse Gaussian process approximations (AISTAT-07)
[**] A distributed variational inference framework for unifying parallel sparse gaussian process regression models (ICML-16)
To summarize, the practical significance of placing such a VI approximation directly on q(f) to encode such (indirectional) local information is, given the above, questionable to me.
Please note that I am not disputing the potential use of this VI form here, which could have been the only way to encode a different (directional) type of information. 
For encoding KNN information, however, it only seems to create more troubles than it solves. 
Minor point: 
The above references, especially [*], should have been cited. 
TECHNICAL SOUNDNESS:
[A] Optimization of the ELBO:
(1) The ordering of data (i.e., the directional information) was mentioned repeatedly in the paper but its importance to the fast approximation was neither explained nor discussed.
(2) The decomposition form of Eq. (6) also raises a question: How do we know that the term inside the log is positive? There seems to be missing information on the constraint of R.
[B] Amortized Inference:
(1) The choice of the GCN seems arbitrary here. I am in fact not sure why GCN is necessary for the inference network & furthermore, GCN also brings to the table another heuristic choice of A.
(2) How do we set the adjacency graph A? 
(3) How do we know what is the right complexity for the GCN?
[C] Complexity: 
The complexity analysis is too informal and lacking fine-grained information. 
Please include a detailed complexity analysis of the training and inference cost in terms of the input dimension, the no. of data points, the size of the neighborhood and the batch size.
It is also necessary to factor in the KNN overhead (e.g., the cost of building the K-D tree for low-dimensional embedding of data & the approximation cost of projecting that information to high-dimensional data)
EXPERIMENT:
The experiment results only show marginal improvement over the baselines, and the size of the dataset for regression is too small. If I read correctly, both have fewer than 20000 data points.
SVGP in particular has been tested on a much larger datasets (AIRLINE, UK Housing) comprising millions of data points -- comparison on such dataset should have been reported. 
Note that the largest dataset used to evaluate the efficiency of fast approximation of GP is on the scale of 6M data points [****]. On that note, eBird and precipitation should not even be considered mid-sized.
To demonstrate the efficiency of local information encoding, comparison with [*] should be reported. There is another class of inducing-point methods that use expectation propagation
that should have been discussed and/or compared with:
[***] A Unifying Framework for Gaussian Process Pseudo-Point Approximations using Power Expectation Propagation (JMLR-18)
[****] Distributed Gaussian Processes (ICML-15)
CLARITY:
The paper is clearly written. 
REVIEW SUMMARY:
This paper adopts a VI approximation that deviates from the conventional form of (Titsias, 2009) to encode the KNN information, which causes extra computational issues (that incurs extra approximations). I find this deviation redundant seeing that the same information could have also been accounted for using the old VI form, which is a lot more computational efficient. I also find the experiment lacking as comparison with fast approximation method such as [*] that incorporate local information is not included. There are also a few technical ambiguities that need to be clarified.
------ Post-Rebuttal Update ------
Thank you for the rebuttal & I have read it in detail. However, it still does not address my concern, which I re-summarize here:
I do not dispute the beneficial of exploiting neighborhood information but my point is we could still leverage the same amount of neighborhood information without going through the trouble of incurring extra steps of approximation due to approximating q(f) instead of q(f_I) -- I think I have elaborated this in points (1) - (4) in my original review -- which also creates the amortized gap. I am also not sure what model reuse means (in this context) and whether it is relevant since it appears somewhat noncentric to the objective of this paper. 
Also, apparently, the experiment is still lacking since to me, comparing with [*] is important in substantiating the contribution claim of this paper.