SilVar-Med: A Speech-Driven Visual Language Model for Explainable
  Abnormality Detection in Medical Imaging
Medical Visual Language Models have shown great potential in various
healthcare applications, including medical image captioning and diagnostic
assistance. However, most existing models rely on text-based instructions,
limiting their usability in real-world clinical environments especially in
scenarios such as surgery, text-based interaction is often impractical for
physicians. In addition, current medical image analysis models typically lack
comprehensive reasoning behind their predictions, which reduces their
reliability for clinical decision-making. Given that medical diagnosis errors
can have life-changing consequences, there is a critical need for interpretable
and rational medical assistance. To address these challenges, we introduce an
end-to-end speech-driven medical VLM, SilVar-Med, a multimodal medical image
assistant that integrates speech interaction with VLMs, pioneering the task of
voice-based communication for medical image analysis. In addition, we focus on
the interpretation of the reasoning behind each prediction of medical
abnormalities with a proposed reasoning dataset. Through extensive experiments,
we demonstrate a proof-of-concept study for reasoning-driven medical image
interpretation with end-to-end speech interaction. We believe this work will
advance the field of medical AI by fostering more transparent, interactive, and
clinically viable diagnostic support systems. Our code and dataset are publicly
available at SiVar-Med.
6807ac6258cd41b34071c428
Tan-Hanh Pham
claimed_verified
2025-05-04T10:05:30.823Z
6654fa747473f3c2572d2f8e
/avatars/8f7c1ef9865f88d538b3f377e1eced0d.svg
Tan-Hanh Pham
user
Hanhpt23
6807ac6258cd41b34071c429
Chris Ngo
claimed_verified
2025-04-23T08:36:51.398Z
63105f7463b70252b4775783
/avatars/89cfe71302cde6c2834f58d44bbcdbd5.svg
Chris Ngo
user
tnngo2
6807ac6258cd41b34071c42a
Trong-Duong Bui
6807ac6258cd41b34071c42b
Minh Luu Quang
6807ac6258cd41b34071c42c
Tan-Huong Pham
6807ac6258cd41b34071c42d
Truong-Son Hy
claimed_verified
2025-05-12T18:34:57.982Z
68221c9b9f55842b8f6e0c31
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/ov4_FLLaOc5ifehP-5i0i.jpeg
Truong-Son Hy
user
hytruongson
SilVar-Med: A Speech-Driven Visual Language Model for Explainable
  Abnormality Detection in Medical Imaging
Medical Visual Language Models have shown great potential in various
healthcare applications, including medical image captioning and diagnostic
assistance. However, most existing models rely on text-based instructions,
limiting their usability in real-world clinical environments especially in
scenarios such as surgery, text-based interaction is often impractical for
physicians. In addition, current medical image analysis models typically lack
comprehensive reasoning behind their predictions, which reduces their
reliability for clinical decision-making. Given that medical diagnosis errors
can have life-changing consequences, there is a critical need for interpretable
and rational medical assistance. To address these challenges, we introduce an
end-to-end speech-driven medical VLM, SilVar-Med, a multimodal medical image
assistant that integrates speech interaction with VLMs, pioneering the task of
voice-based communication for medical image analysis. In addition, we focus on
the interpretation of the reasoning behind each prediction of medical
abnormalities with a proposed reasoning dataset. Through extensive experiments,
we demonstrate a proof-of-concept study for reasoning-driven medical image
interpretation with end-to-end speech interaction. We believe this work will
advance the field of medical AI by fostering more transparent, interactive, and
clinically viable diagnostic support systems. Our code and dataset are publicly
available at SiVar-Med.
SilVar-Med is an end-to-end speech-driven medical visual language model that enhances interpretability and interaction in medical image analysis through voice commands and reasoning-driven predictions.
speech-driven medical visual language models multimodal medical image assistant voice-based communication medical image analysis reasoning-driven medical image interpretation