Despite the eminent successes of deep neural networks, many architectures are often hard to transfer to irregularly-sampled and asynchronous time series that occur in many real-world datasets, such as healthcare applications. This paper proposes a novel framework for classifying irregularly sampled time series with unaligned measurements, focusing on high scalability and data efficiency. Our method SEFT (Set Functions for Time Series) is based on recent advances in differentiable set function learning, extremely parallelizable, and scales well to very large datasets and online monitoring scenarios. We extensively compare our method to competitors on multiple healthcare time series datasets and show that it performs competitively whilst significantly reducing runtime.
Summary:
The work is focused on classification of irregularly sampled and unaligned multi-modal time series. Prior work has primarily focused on imputation methods, either end-to-end or otherwise. This paper approaches the problem as a set function mapping between the time-series tuples to the class label. The proposed method is uses a set encoding of a multi-modal time series input, followed by mode-specific encoding of the tuples which are then aggregated in multiple ways prior to classification. An attention mechanism is attached in order for the model to automatically weigh the relevance of tuples for the classification. The model is compared to imputation based baselines on clinical ICU time series classification tasks. The performance mostly appears comparable across baselines but the proposed method has much better run-times. 
The paper is for the most part well written, and related work well characterized. The formulation is interesting and clinically relevant as well so the choice of data-sets makes some sense. I have a few concerns about the architecture formulation and lack of clarification and intuition in what appears to be the main contribution of the paper (Sec 3.2 and 3.3) which I will detail below:
a. In the evaluation, I really want to see a decoupling between the "time encoding step" and "attention based aggregation" on the performance to figure out to isolate different sources of performance improvements. That is can there be a SEFT without time encoding? If not, why not? I encourage more ablation like studies that look at different sources of performance gains and demonstrate them in experiments.
b. The description of Sec 3.3. is really missing key motivation for the choices made around how the attention formulation is designed. For example why does the dot produce include the set elements? What if it doesn't? What is Q supposed to capture? 
c. Is a_{j,i} shared across instances? Then irrespective of the number of observations per instance, the $j^{th}$ tuple gets similar weights? If not appropriate indexing will help clarify this.
d. It would be useful to provide how exactly a label is inferred for a *new* test instance. 
I have some minor additional feedback (just for presentation and motivation purposes):
1. Authors make a claim in the introduction which should likely be qualified with a citation - "Furthermore, even though a decoupled imputation scheme followed by classification is generally more scalable, it may lose information that is relevant for prediction tasks". How does decoupled imputation imply loss of relevant information? By losing information about which observations are missing and relying on that for prediction? Does this clinically make sense? Or even generally? 
2. In Sec 3.3, you probably mean $W_i \in R^{(im(f') + |s_j|) \times d}$. That is parenthesis are missing?
3. What are the +- std errors indicating? Is it cross validation error on a held-out test set? 
4. Initially $i$ is indexing samples and by equation (3), (4) $i$ indexes time(?) and in Sec 3.3 $i$ indexes observations? How are observations defined here? is it measurement of specific modality at a specific time instance? Can you clear this in the introduction itself? 
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
I have read the authors updated draft and response. The experiments section looks much better now. 
1. The overall contribution has less clinical utility in my opinion as generally a patient likely deteriorates over time before an adverse outcome and therefore -- to give the model too much flexibility w.r.t. time ordering doesn't make quite as much sense. This is reflected in the fact that experimental results are not drastically better than other baselines. The authors might be able to show the utility of the method on other time series classification datasets where this is not a limitation of the data itself. However in those settings, it may be a bit hard to beat transformers. Do the authors have a sense of where the benefits of this method really are?
2. Mortality tasks are generally on the simpler side of clinical prediction problems as well. Nonetheless I think the contribution has some utility to the community. I do encourage the authors to try non--clinical datasets for a comparison
3. Please have a discussion that includes limitations and to discuss where the benefits of your methods really lie. A clear and thoughtful discussion is currently missing in your conclusions. 
With that said, I am updating my score to a 6.