Convolutional neural network (CNN) for time series data implicitly assumes that the data are uniformly sampled, whereas many event-based and multi-modal data are nonuniform or have heterogeneous sampling rates. Directly applying regular CNN to nonuniform time series is ungrounded because it is unable to recognize and extract common patterns from the nonuniform input signals. Converting the nonuniform time series to uniform ones by interpolation preserves the pattern extraction capability of CNN, but the interpolation kernels are often preset and may be unsuitable for the data or tasks. In this paper, we propose the Continuous CNN (CCNN), which estimates the inherent continuous inputs by interpolation, and performs continuous convolution on the continuous input. The interpolation and convolution kernels are learned in an end-to-end manner, and are able to learn useful patterns despite the nonuniform sampling rate. Besides, CCNN is a strict generalization to CNN. Results of several experiments verify that CCNN achieves a better performance on nonuniform data, and learns meaningful continuous kernels.
The paper proposes a continuous CNN model to accommodate the nonuniform time series data. The model learns the interpolation and the convolution kernel functions in an end-to-end manner, so that it can capture the signal patterns and be flexible. A layer has three networks, which learn a kernel function to represent the combination of interpolation and convolution, a bias function to represent the error correction with convolution, and then produce the output based on them. The authors introduce two assumptions and a two-hot encoding scheme for the input to control the model complexity. The paper also introduces an application of the proposed CCNN by combing with temporal point process. Experiments on simulated data compares the proposed method with some degenerative baselines show the advantage of learning the interpolation and the two-hot encoding configuration. The authors compare the performance on time interval prediction task based on real world dataset to show the model produces a better history embedding for the task.
Overall, the paper has some incremental improvements on the existing methods that dealing with the nonuniform time series data. Instead of using preset interpolation kernels, the proposed model can learn it with the convolution in a data-driven manner.
The paper includes clear explanation on module structure and detailed experiment settings.
The experiments of signal value prediction support the claims of the advantages of the proposed model.
The notation in the caption of figure 1 is a little confusing: is x(t_i’) the same as hat x(t) in the algorithm?
It is good that the related works section mentioned the adapted RNNs that are used as baselines in the real-world dataset experiment, and the differences between the proposed model and the related SNNs are introduced.
However, this section and the introduction can be better organized to distinguish the novelty and the contribution of the work.
In page 7, the purpose of the reference in the sentence “The time information is either two-hot encoded (Adams et al., 2010) (CCNN-th), or not encoded (CCNN).” is not very clear.
It will be better if there are a little more analysis of the experiment on predicting time intervals to next event.
The upper plots in the figure may be not convincing enough to support the claims.
The advantage of two-hot encoding seems subtle in the figure 5. Is there any reason for the significantly higher deviation of CCNN-th in StackOverflow?
Sometimes the usage of “CCNN” is not clear, for example, the experiment on speech interpolation compares the “CCNN-th” method with baselines, but uses “CCNN” in the analysis. Also, it could be better to show the “CCNN-th” result in the upper plots of figure 5 instead of “CCNN”.
Minor comment:
There are some typos in the paper, for example, missing the right parenthesis in page3 “(refer to Appendix A.1”, in page4 section 4.1 “According to Eq.(4), the input is …”.
“The left plot shows” in the last line of the caption of figure 3 should be “right”.