How to evaluate your medical time series classification?
<|reference_start|>How to evaluate your medical time series classification?: Medical time series (MedTS) play a critical role in many healthcare applications, such as vital sign monitoring and the diagnosis of brain and heart diseases. However, the existence of subject-specific features poses unique challenges in MedTS evaluation. Inappropriate evaluation setups that either exploit or overlook these features can lead to artificially inflated classification performance (by up to 50% in accuracy on ADFTD dataset): this concern has received little attention in current research. Here, we categorize the existing evaluation setups into two primary categories: subject-dependent and subject-independent. We show the subject-independent setup is more appropriate for different datasets and tasks. Our theoretical analysis explores the feature components of MedTS, examining how different evaluation setups influence the features that a model learns. Through experiments on six datasets (spanning EEG, ECG, and fNIRS modalities) using four different methods, we demonstrate step-by-step how subject-dependent utilizes subject-specific features as a shortcut for classification and leads to a deceptive high performance, suggesting that the subject-independent setup is more precise and practicable evaluation setup in real-world. This comprehensive analysis aims to establish clearer guidelines for evaluating MedTS models in different healthcare applications. Code to reproduce this work in \url{https://github.com/DL4mHealth/MedTS_Evaluation}.<|reference_end|>
@article{wang2024how,
    title={How to evaluate your medical time series classification?},
    author={Yihe Wang, Taida Li, Yujun Yan, Wenzhan Song, Xiang Zhang},
    journal={arXiv preprint arXiv:2410.03057},
    year={2024},
    archivePrefix={arXiv},
    eprint={2410.03057},
    primaryClass={cs.CE}
}