This paper introduces CloudLSTM, a new branch of recurrent neural models tailored to forecasting over data streams generated by geospatial point-cloud sources. We design a Dynamic Point-cloud Convolution (DConv) operator as the core component of CloudLSTMs, which performs convolution directly over point-clouds and extracts local spatial features from sets of neighboring points that surround different elements of the input. This operator maintains the permutation invariance of sequence-to-sequence learning frameworks, while representing neighboring correlations at each time step -an important aspect in spatiotemporal predictive learning. The DConv operator resolves the grid-structural data requirements of existing spatiotemporal forecasting models and can be easily plugged into traditional LSTM architectures with sequence-to-sequence learning and attention mechanisms. We apply our proposed architecture to two representative, practical use cases that involve point-cloud streams, i.e., mobile service traffic forecasting and air quality indicator forecasting. Our results, obtained with real-world datasets collected in diverse scenarios for each use case, show that CloudLSTM delivers accurate long-term predictions, outperforming a variety of neural network models.
Review Summary
--------------
Overall this is almost above the bar for me to accept, but I think there's enough concerns about the method and experiments that I'm hesitant. Strengths include the invariance to point cloud order and the relative simplicity of the architecture (compared to PointCNN). Weaknesses include a vulnerability to outliers, experiments that don't seem to think about practical effects like day-of-week in forecasting, and experiments that leave out baselines to help directly assess the impact of neighbors.
Paper Summary
-------------
The paper develops a new neural net architecture for processing data structured as spatial point clouds that vary over time (e.g. hourly traffic at several antennas spread throughout a city).
The core of the approach is a new neural net unit: the "D-Conv" operator (See Eq. 2). The output value at each point is obtained via a weighted combination of nearby coordinates and features, using only the K-nearest neighbors (stored in ranked order) to maintain invariance to the original order of points. This layer can be included in modern convolutional (CloudCNN) or recurrent (CloudLSTM) or attention-based architectures in a straightforward way.
Unlike many previous methods that require converting point clouds to quantized regular grids, the present approach directly consumes point cloud data. Unlike some existing methods like PointCNN, it avoids information loss (does not reduce dimension from input to output layer).
Two experimental evaluations are conducted: forecasting mobile app traffic across 2 European cities (given past 30 min, predict next 30 min), and air quality across several regions in China (given last 12 hrs, predict next 12 hrs). In both experiments, the locations of the sensors are fixed across time. Fig 4 further looks at traffic forecasting as a function of the lookahead time, from 0-3 hours ahead.
Novelty & Significance
-----------------------
The paper definitely tackles an important problem (point cloud forecasting). 
The present paper's new "D-Conv" operator appears new, though it looks really like a simplification of the PointCNN's "X-Conv" operator rather than a brand new operator. 
The most similar work seems to be the PointCNN (Li et al NeurIPS 2018). This work's contribution was a new "X-Conv" operator, which also consumes point clouds and produces learned representations. X-Conv, like the present paper's D-Conv, computes K-nearest neighbors of each point p, but performs first an embedding of each neighbor to a learned "local" feature space and then performs convolution on this embedding. Perhaps the biggest practical difference is that D-Conv has fewer parameters (does not perform the embedding) and does not reduce dimensionality from input to output. 
Technical Concerns
------------------
My biggest concerns are that the D-Conv has a strong reliance on nearest neighbors. This means the D-Conv has not much accomodation for "outlier" points that are far from others. The X-Conv operator has some nice properties in this regard (it changes coordinate systems so neighbor locations are centered around the current point), but I don't see this in the D-Conv operator, as in Eq. 2, where the coordinate locations are fed directly into the weighted sum after global rescaling to (0,1). I would imagine that data with outliers (whose values are unlike most others) would dramatically hurt performance, as the weights of D-Conv would need to be shared equally by outliers and inliers.
Experimental Concerns
---------------------
Is there a good reason to not try to compare on publicly available datasets like those used in the PointCNN paper (focusing only on the non-temporal versions of the model)? Using proprietary datasets makes following up on this work a bit hard, would be nice to have some reproducible experiment.
It's not clear to me that the experiments here consider realistic scenarios. Why would I predict mobile app traffic using only the past 30 minutes of data? Why predict air quality using only the last 12 hours? Certainly there are time-of-day, day-of-week, and seasonal effects that are all important. At a minimum, I'd think that for the mobile traffic case you could at least look at consuming the last 48 hr of data and predicting the next 30-90 minutes. I suspect that would make even simpler models do much better.  
Further, I think the experiments are missing some key simple baselines (or I misunderstand something). For example, rather than the complicated CNN/LSTM architectures, why not try to directly see how much value there is in "neighbors" in this 2d space? At each point, you can make predictions using only the K nearest neighbors' data, with K swept from 1 to 100 or something. I would expect with these features, using just a simple MLP or RNN would do quite well. I'd like to see a stronger qualitative case made for why we expect the complicated DConv weighting operator here to do better than this baselines.
Overall, the results tables appear promising (for app traffic forecasting in Table 1, the proposed CloudLSTM achieves 3.66 MAE compared to 4.95 for PointCNN and 4.8 for an MLP). However, it's not clear why and I'd like to understand why. Is it that the other approaches are overfitting? 
Minor Concerns
--------------
I would suggest avoiding calling the method "\mathcal{D}-Conv", and instead use just "DConv", since this is easier to type into search engines and easier to search for in a PDF document
Related: Point clouds could be represented as graphs, and then use graph embeddings as feature representations