RISE and DISE: Two Frameworks for Learning from Time Series with Missing Data
Time series with missing data constitute an important setting for machine learning. The most successful prior approaches for modeling such time series are based on recurrent neural networks that learn to impute unobserved values and then treat the imputed values as observed. We start by introducing Recursive Input and State Estimation (RISE), a general framework that encompasses such prior approaches as specific instances. Since RISE instances tend to suffer from poor long-term performance as errors are amplified in feedback loops, we propose Direct Input and State Estimation (DISE), a novel framework in which input and state representations are learned from observed data only. The key to DISE is to include time information in representation learning, which enables the direct modeling of arbitrary future time steps by effectively skipping over missing values, rather than imputing them, thus overcoming the error amplification encountered by RISE methods. We benchmark instances of both frameworks on two forecasting tasks, observing that DISE achieves state-of-the-art performance on both.
Overall: Provides a nice summary of different methods for dealing with missing values in neural net time series models and proposes a new technique that does not involve running a series of possibly diverging predictions but rather jumps ahead to reason about arbitrary points in the future in a “single hop”, avoiding the risks associated with compounding errors. Also proposes a new method for encoding values that’s quite unusual but appears to work very well. Overall, the paper is mostly clear, the technique is reasonable, and the best model does indeed appear to work well. I have only one serious reservation about this paper - and it is an extremely serious concern about the experimental setup, and I would ask that the authors clarify this point for me in a response. DISE works poorly or only comparably well to the baselines in all tasks unless the GRU-based input encoding is used. Obviously any RISE model likewise depends on an input encoding, so the question is whether the baseline RISE models were given the benefit of the GRU-based input encoding. If not, please provide this comparison. Minor comments: “replace the standard input with a transformed input ˆx” -> I find this wording awkward. If the input is missing, it cannot be transformed, it can be predicted using a conditional model over data given a representation of past (and/or future) observations, or it can be a (probably learned) dummy value, but please clarify this wording- it’s essential. Similarity of DISE to prior work: There are a number of processes that build representations based on measurements that happen at random times without “rolling forward” a single step model, for instance, the neural Hawkes process (Mei and Eisner, 2017 or so), which has also been applied to impute missing values. Some discussion of the relationship to work like this is recommended. Additionally, the idea of learning representations based on predicting values at several time scales into the future comes up in contrastive predictive coding (van Oord et al, 2018).
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
RISE and DISE: Two Frameworks for Learning from Time Series with Missing Data
Time series with missing data constitute an important setting for machine learning. The most successful prior approaches for modeling such time series are based on recurrent neural networks that learn to impute unobserved values and then treat the imputed values as observed. We start by introducing Recursive Input and State Estimation (RISE), a general framework that encompasses such prior approaches as specific instances. Since RISE instances tend to suffer from poor long-term performance as errors are amplified in feedback loops, we propose Direct Input and State Estimation (DISE), a novel framework in which input and state representations are learned from observed data only. The key to DISE is to include time information in representation learning, which enables the direct modeling of arbitrary future time steps by effectively skipping over missing values, rather than imputing them, thus overcoming the error amplification encountered by RISE methods. We benchmark instances of both frameworks on two forecasting tasks, observing that DISE achieves state-of-the-art performance on both.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



Overall: Provides a nice summary of different methods for dealing with missing values in neural net time series models and proposes a new technique that does not involve running a series of possibly diverging predictions but rather jumps ahead to reason about arbitrary points in the future in a “single hop”, avoiding the risks associated with compounding errors. Also proposes a new method for encoding values that’s quite unusual but appears to work very well. Overall, the paper is mostly clear, the technique is reasonable, and the best model does indeed appear to work well. I have only one serious reservation about this paper - and it is an extremely serious concern about the experimental setup, and I would ask that the authors clarify this point for me in a response. DISE works poorly or only comparably well to the baselines in all tasks unless the GRU-based input encoding is used. Obviously any RISE model likewise depends on an input encoding, so the question is whether the baseline RISE models were given the benefit of the GRU-based input encoding. If not, please provide this comparison. Minor comments: “replace the standard input with a transformed input ˆx” -> I find this wording awkward. If the input is missing, it cannot be transformed, it can be predicted using a conditional model over data given a representation of past (and/or future) observations, or it can be a (probably learned) dummy value, but please clarify this wording- it’s essential. Similarity of DISE to prior work: There are a number of processes that build representations based on measurements that happen at random times without “rolling forward” a single step model, for instance, the neural Hawkes process (Mei and Eisner, 2017 or so), which has also been applied to impute missing values. Some discussion of the relationship to work like this is recommended. Additionally, the idea of learning representations based on predicting values at several time scales into the future comes up in contrastive predictive coding (van Oord et al, 2018).
assistant
Overall: Provides a nice summary of different methods for dealing with missing values in neural net time series models and proposes a new technique that does not involve running a series of possibly diverging predictions but rather jumps ahead to reason about arbitrary points in the future in a “single hop”, avoiding the risks associated with compounding errors. Also proposes a new method for encoding values that’s quite unusual but appears to work very well. Overall, the paper is mostly clear, the technique is reasonable, and the best model does indeed appear to work well. I have only one serious reservation about this paper - and it is an extremely serious concern about the experimental setup, and I would ask that the authors clarify this point for me in a response. DISE works poorly or only comparably well to the baselines in all tasks unless the GRU-based input encoding is used. Obviously any RISE model likewise depends on an input encoding, so the question is whether the baseline RISE models were given the benefit of the GRU-based input encoding. If not, please provide this comparison. Minor comments: “replace the standard input with a transformed input ˆx” -> I find this wording awkward. If the input is missing, it cannot be transformed, it can be predicted using a conditional model over data given a representation of past (and/or future) observations, or it can be a (probably learned) dummy value, but please clarify this wording- it’s essential. Similarity of DISE to prior work: There are a number of processes that build representations based on measurements that happen at random times without “rolling forward” a single step model, for instance, the neural Hawkes process (Mei and Eisner, 2017 or so), which has also been applied to impute missing values. Some discussion of the relationship to work like this is recommended. Additionally, the idea of learning representations based on predicting values at several time scales into the future comes up in contrastive predictive coding (van Oord et al, 2018).