ABSTRACT Self-supervision, in which a target task is improved without external supervision, has primarily been explored in settings that assume the availability of additional data. However, in many cases, particularly in healthcare, one may not have access to additional data (labeled or otherwise). In such settings, we hypothesize that self-supervision based solely on the structure of the data at-hand can help. We explore a novel self-supervision framework for time-series data, in which multiple auxiliary tasks (e.g., forecasting) are included to improve overall performance on a sequence-level target task without additional training data. We call this approach limited self-supervision, as we limit ourselves to only the data at-hand. We demonstrate the utility of limited self-supervision on three sequence-level classification tasks, two pertaining to real clinical data and one using synthetic data. Within this framework, we introduce novel forms of self-supervision and demonstrate their utility in improving performance on the target task. Our results indicate that limited self-supervision leads to a consistent improvement over a supervised baseline, across a range of domains. In particular, for the task of identifying atrial fibrillation from small amounts of electrocardiogram data, we observe a nearly 13% improvement in the area under the receiver operating characteristics curve (AUC-ROC) relative to the baseline (AUC-ROC=0.55 vs. AUC-ROC=0.62). Limited self-supervision applied to sequential data can aid in learning intermediate representations, making it particularly applicable in settings where data collection is difficult.
This paper propose an approach for self-supervised learning on time series.
Three datasets are considered (simulation and 2 healthcare datasets).
The gist of the contribution is to both optimize prediction loss
of the true task and at the same time do a good job for a family
of auxiliary tasks. 4 auxiliary tasks are considered. While
the first 3 auxiliary tasks are quite common, the 4th tasks
called piecewise-linear autoencoding appears novel. The idea
is that the hidden representation of the LSTM should be a good predictor
of the past using a piecewise-linear approximation.
The author coin the term "limited self-supervision" for their approach
although it's not clear why it is fundamentally not just self-supervised
learning as it has been proposed in the past.
The paper is overall well written and addresses the relevant issue
of learning from limited annotated data.
Major concerns
- It is yet another way to do self-supervised learning on time series
and no clear benchmark with alternatives is provided (time contrastive
learning (TCL) or Contrastive Predictive Coding (CPC) https://arxiv.org/pdf/1807.03748.pdf
etc.)
- On any of the applied problem it is not clear if the proposed
approach brings an improvement on the state-of-the-art or if it's
just an illustration of the method disconnected from the literature
of the application.