Published as a conference paper at ICLR 2018 AUTOMATICALLY INFERRING DATA QUALITY FOR SPATIOTEMPORAL FORECASTING
Spatiotemporal forecasting has become an increasingly important prediction task in machine learning and statistics due to its vast applications, such as climate modeling, traffic prediction, video caching predictions, and so on. While numerous studies have been conducted, most existing works assume that the data from different sources or across different locations are equally reliable. Due to cost, accessibility, or other factors, it is inevitable that the data quality could vary, which introduces significant biases into the model and leads to unreliable prediction results. The problem could be exacerbated in black-box prediction models, such as deep neural networks. In this paper, we propose a novel solution that can automatically infer data quality levels of different sources through local variations of spatiotemporal signals without explicit labels. Furthermore, we integrate the estimate of data quality level with graph convolutional networks to exploit their efficient structures. We evaluate our proposed method on forecasting temperatures in Los Angeles.
The paper is an application of neural nets to data quality assessment. The authors introduce a new definition of data quality that relies on the notion of local variation defined in (Zhou and Schölkopf, 2004), and they extend it to multiple heterogenous data sources. The data quality function is learned using a GCN as defined in (Kipf and Welling, 2016).
 
1) How many neighbors are used in the experiments? Is this fixed or is defined purely by the Gaussian kernel weights as mentioned in 4.2? Setting all weights less than 0.9 to zero seems quite abrupt. Could you provide a reason for this? How many neighbors fit in this range?
2) How many data points? What is the temporal resolution of your data (every day/hour/minute/etc.)? What is the value of N, T? 
3) The bandwidth of the Gaussian kernel (\gamma) is quite different (0.2 and 0.6) for the two datasets from Weather Underground (WU) and Weather Bug (WB) (sect. 4.2). The kernel is computed on the features (e.g., latitude, longitude, vegetation fraction, etc.). Location, longitude, distance from coast, etc. are the same no matter the data source (WU or WB). Maybe the way they compute other features (e.g., vegetation fraction) vary slightly, but I would guess the \gamma should be (roughly) the same. 
4) Are the features (e.g., latitude, longitude, vegetation fraction, etc.) normalized in the kernel? If they are given equal weight (which is in itself questionable) they should be normalized, otherwise some of them will always dominate the distances. If they were indeed normalized, that should be made clear in the paper.
5) Why do you choose the summer months? How does the framework perform on other months and other meteorological signals except temperature? The application is very nice and complex, but I find that the experiments are a little bit too limited. 
6) The notations w and W are used for different things, and that is slightly confusing. Usually one is used as a matrix notation of the other.
7) I would tend to associate data quality with how noisy observations are at a certain node, and not heterogeneity. It would be good to add some discussion on noise in the paper. How do you define an anomaly in 5.2? Is it a spatial or temporal anomaly? Not sure I understand the difference between an anomaly and a bridge node. 
8) “A bridge node highly likely has a lower data quality level due to the heterogeneity”. I would think a bridge node is very relevant, and I don't necessarily see it as having a lower data quality. This approach seems to give more weight to very similar data, while discarding the transitions, which in a meteorological setting, could be the most relevant ?! 
9) Update the references, some papers have updated information (e.g., Bruna et al. - ICLR 2014, Kipf and Welling – ICLR 2017, etc.).
Quality – The experiments need more work and editing as mentioned in the comments.  
Clarity – The framework is fairly clearly presented, however the English needs significant improvement. 
Originality – The paper is a nice application of machine learning and neural nets to data quality assessment, and the forecasting application is relevant and challenging. However the paper mostly provides a framework that relies on existing work.
Significance – While the paper could be relevant to data quality applications by introducing advanced machine learning techniques, it has limited reach outside the field. Maybe publish it in a data quality conference/journal.