Self-supervised Multimodal Speech Representations for the Assessment of Schizophrenia Symptoms
<|reference_start|>Self-supervised Multimodal Speech Representations for the Assessment of Schizophrenia Symptoms: Multimodal schizophrenia assessment systems have gained traction over the last few years. This work introduces a schizophrenia assessment system to discern between prominent symptom classes of schizophrenia and predict an overall schizophrenia severity score. We develop a Vector Quantized Variational Auto-Encoder (VQ-VAE) based Multimodal Representation Learning (MRL) model to produce task-agnostic speech representations from vocal Tract Variables (TVs) and Facial Action Units (FAUs). These representations are then used in a Multi-Task Learning (MTL) based downstream prediction model to obtain class labels and an overall severity score. The proposed framework outperforms the previous works on the multi-class classification task across all evaluation metrics (Weighted F1 score, AUC-ROC score, and Weighted Accuracy). Additionally, it estimates the schizophrenia severity score, a task not addressed by earlier approaches.<|reference_end|>
@article{premananth2024self-supervised,
    title={Self-supervised Multimodal Speech Representations for the Assessment of
  Schizophrenia Symptoms},
    author={Gowtham Premananth and Carol Espy-Wilson},
    journal={arXiv preprint arXiv:2409.09733},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.09733},
    primaryClass={eess.AS cs.SD eess.SP}
}