In health, machine learning is increasingly common, yet neural network embedding (representation) learning is arguably under-utilized for physiological signals. This inadequacy stands out in stark contrast to more traditional computer science domains, such as computer vision (CV), and natural language processing (NLP). For physiological signals, learning feature embeddings is a natural solution to data insufficiency caused by patient privacy concerns -rather than share data, researchers may share informative embedding models (i.e., representation models), which map patient data to an output embedding. Here, we present the PHASE (PHysiologicAl Signal Embeddings) framework, which consists of three components: i) learning neural network embeddings of physiological signals, ii) predicting outcomes based on the learned embedding, and iii) interpreting the prediction results by estimating feature attributions in the "stacked" models (i.e., feature embedding model followed by prediction model). PHASE is novel in three ways: 1) To our knowledge, PHASE is the first instance of transferal of neural networks to create physiological signal embeddings. 2) We present a tractable method to obtain feature attributions through stacked models. We prove that our stacked model attributions can approximate Shapley values -attributions known to have desirable properties -for arbitrary sets of models. 3) PHASE was extensively tested in a cross-hospital setting including publicly available data. In our experiments, we show that PHASE significantly outperforms alternative embeddingssuch as raw, exponential moving average/variance, and autoencoder -currently in use. Furthermore, we provide evidence that transferring neural network embedding/representation learners between distinct hospitals can yield performant embeddings and offer recommendations when transference is ineffective.
Summary of the paper:
This paper proposes PHASE, a framework to learn the embeddings for physiological signals from medical records, which can be used in downstream prediction tasks, possibly across domains (i.e. different patient distribution). The authors employ separate LSTMs for each signal channel that are trained to predicts the minimum value of the signal in the fixed future time window (5 minutes in this paper). After training the LSTMs, the learned signal embeddings are fed to gradient boosted trees for a specific prediction task (e.g. predicting whether hypoxemia will occur in 5 minutes). Once the LSTMs are trained, they can be re-used for another dataset; the LSMTs are fixed, and generate embeddings that are fed to a new trainable gradient boosted trees for performing a similar task. The authors also combine existing attribution methods (DeepSHAP and Independent TreeSHAP) to provide some explanation of PHASE. The authors use three different datasets to test PHASE's prediction performance, transferability of the embeddings, and interpretation.
Pros:
- The paper is well-motivated, well-organized and clearly written. The reading experience was smooth.
- Given the importance of physiological signals in ICU settings, transferable embeddings can be an important technique in practice
- As the authors claim, I am not aware of any notable prior work on transferable physiological signal embeddings. The authors tackle a relatively unexplored territory.
Issues:
- The authors claim PHASE learns signal embeddings that are transferable. However, the authors train the embeddings to predict the minimum value within the next five time steps, because the downstream tasks are all predicting whether a certain signal goes below some threshold ("hypo"xenia, "hypo"capnia, "hypo"tension). This means the authors designed the embedding learning process with a priori knowledge of the downstream tasks, which significantly weakens theirs claim that PHASE learns transferable embeddings. Word embeddings trained on Wikipedia, or ConvNets trained on ImageNet are not designed to be used in a specific type of downstream tasks. What PHASE demonstrates is basically that "hypo"xxxx predictions can be accurately made with pre-training the embeddings to predict a very relevant task. 
- The authors claim that transferred PHASE embeddings significantly outperform EMA or Raw. But I wouldn't call 0.005-0.02 AP improvement "significant". Model 12 in Figure 3 shows better performance than model 2 and 4, but the gap is not that large.
- More importantly, the fact that model 10 and model 12 show similar performance is not very surprising. The two hospitals are in the same city, only miles away. Naturally the distribution of the patients would not be too different. Given this, claiming that PHASE embeddings are transferable does not have a strong ground.
- The claim for transferable embedding is further weakened by Figure 4. Model 1^p in Figure 4 clearly performs worse than Raw, which means embeddings learned from significantly different setting (hospital P) is actually making it harder for XGB than simply looking at raw signals. If PHASE was learning a robust embeddings, then the learned embeddings should at least not hurt the performance of XGB.
- Evaluating the interpretation of the model is weak. All the authors did was pick four examples and provide qualitative explanation. And they do not even describe whether this interpretation is from model 9 or 10. It would have been much better if at least one medical expert took a look at more than a few examples. In the current form, we cannot be sure if the model is using the SaO2 signal in a medically meaningful way. Also, if this is the interpretation of model 10 or 12, then we should look at the attributions for other signals as well.
- Lack of description on experiment setup. The authors do not describe how they pre-trained the LSTMs to obtain Min^h, Auto^h and Hypox^h, which significantly hurts reproducibility. Also I couldn't find any description regarding train/test splits or cross validations, or size of the LSTM cells.
- More description is necessary as to how Raw was used to train XGB. Was the entire sequence of 15 signals fed to XGB?
- Y-axis of Figure 5 is not on the same scale. This makes it hard to intuitively understand the change of SaO2.