Teach Multimodal LLMs to Comprehend Electrocardiographic Images
The electrocardiogram (ECG) is an essential non-invasive diagnostic tool for
assessing cardiac conditions. Existing automatic interpretation methods suffer
from limited generalizability, focusing on a narrow range of cardiac
conditions, and typically depend on raw physiological signals, which may not be
readily available in resource-limited settings where only printed or digital
ECG images are accessible. Recent advancements in multimodal large language
models (MLLMs) present promising opportunities for addressing these challenges.
However, the application of MLLMs to ECG image interpretation remains
challenging due to the lack of instruction tuning datasets and well-established
ECG image benchmarks for quantitative evaluation. To address these challenges,
we introduce ECGInstruct, a comprehensive ECG image instruction tuning dataset
of over one million samples, covering a wide range of ECG-related tasks from
diverse data sources. Using ECGInstruct, we develop PULSE, an MLLM tailored for
ECG image comprehension. In addition, we curate ECGBench, a new evaluation
benchmark covering four key ECG image interpretation tasks across nine
different datasets. Our experiments show that PULSE sets a new
state-of-the-art, outperforming general MLLMs with an average accuracy
improvement of 15% to 30%. This work highlights the potential of PULSE to
enhance ECG interpretation in clinical practice.
671ee90c77035878c52e67b2
Ruoqi Liu
admin_assigned
2024-10-28T15:18:19.340Z
656fd85f039133e875431ed3
https://cdn-avatars.huggingface.co/v1/production/uploads/656fd85f039133e875431ed3/_Fc8m0hk0MXOfi3470Kru.jpeg
Ruoqi Liu
user
RuoqiLiu
671ee90c77035878c52e67b3
Yuelin Bai
671ee90c77035878c52e67b4
Xiang Yue
admin_assigned
2024-10-28T15:18:37.003Z
6230d750d93e84e233882dbc
https://cdn-avatars.huggingface.co/v1/production/uploads/6230d750d93e84e233882dbc/4MGEekLW3oWzqeFWDWvIK.jpeg
Xiang Yue
user
yuexiang96
671ee90c77035878c52e67b5
Ping Zhang
admin_assigned
2024-10-28T15:19:05.468Z
671f96c070f231ff42ceadc6
https://cdn-avatars.huggingface.co/v1/production/uploads/671f96c070f231ff42ceadc6/lIRWnc9-zItr6QoGs0N3u.jpeg
Ping Zhang
user
aidhlab
Teach Multimodal LLMs to Comprehend Electrocardiographic Images
The electrocardiogram (ECG) is an essential non-invasive diagnostic tool for
assessing cardiac conditions. Existing automatic interpretation methods suffer
from limited generalizability, focusing on a narrow range of cardiac
conditions, and typically depend on raw physiological signals, which may not be
readily available in resource-limited settings where only printed or digital
ECG images are accessible. Recent advancements in multimodal large language
models (MLLMs) present promising opportunities for addressing these challenges.
However, the application of MLLMs to ECG image interpretation remains
challenging due to the lack of instruction tuning datasets and well-established
ECG image benchmarks for quantitative evaluation. To address these challenges,
we introduce ECGInstruct, a comprehensive ECG image instruction tuning dataset
of over one million samples, covering a wide range of ECG-related tasks from
diverse data sources. Using ECGInstruct, we develop PULSE, an MLLM tailored for
ECG image comprehension. In addition, we curate ECGBench, a new evaluation
benchmark covering four key ECG image interpretation tasks across nine
different datasets. Our experiments show that PULSE sets a new
state-of-the-art, outperforming general MLLMs with an average accuracy
improvement of 15% to 30%. This work highlights the potential of PULSE to
enhance ECG interpretation in clinical practice.
ECGInstruct and ECGBench improve ECG image interpretation using a specialized multimodal large language model, PULSE, which outperforms general models.
multimodal large language models MLLMs ECG image instruction tuning dataset ECG image comprehension evaluation benchmark ECG image interpretation tasks