title: SilVar-Med: A Speech-Driven Visual Language Model for Explainable
    Abnormality Detection in Medical Imaging
summary: Medical Visual Language Models have shown great potential in various
  healthcare applications, including medical image captioning and diagnostic
  assistance. However, most existing models rely on text-based instructions,
  limiting their usability in real-world clinical environments especially in
  scenarios such as surgery, text-based interaction is often impractical for
  physicians. In addition, current medical image analysis models typically lack
  comprehensive reasoning behind their predictions, which reduces their
  reliability for clinical decision-making. Given that medical diagnosis errors
  can have life-changing consequences, there is a critical need for interpretable
  and rational medical assistance. To address these challenges, we introduce an
  end-to-end speech-driven medical VLM, SilVar-Med, a multimodal medical image
  assistant that integrates speech interaction with VLMs, pioneering the task of
  voice-based communication for medical image analysis. In addition, we focus on
  the interpretation of the reasoning behind each prediction of medical
  abnormalities with a proposed reasoning dataset. Through extensive experiments,
  we demonstrate a proof-of-concept study for reasoning-driven medical image
  interpretation with end-to-end speech interaction. We believe this work will
  advance the field of medical AI by fostering more transparent, interactive, and
  clinically viable diagnostic support systems. Our code and dataset are publicly
  available at SiVar-Med.
paper_authors._id: 6807ac6258cd41b34071c428
paper_authors.name: Tan-Hanh Pham
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-04T10:05:30.823Z
paper_authors._id: 6654fa747473f3c2572d2f8e
paper_authors.avatarUrl: /avatars/8f7c1ef9865f88d538b3f377e1eced0d.svg
paper_authors.fullname: Tan-Hanh Pham
paper_authors.type: user
paper_authors.user: Hanhpt23
paper_authors._id: 6807ac6258cd41b34071c429
paper_authors.name: Chris Ngo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-23T08:36:51.398Z
paper_authors._id: 63105f7463b70252b4775783
paper_authors.avatarUrl: /avatars/89cfe71302cde6c2834f58d44bbcdbd5.svg
paper_authors.fullname: Chris Ngo
paper_authors.type: user
paper_authors.user: tnngo2
paper_authors._id: 6807ac6258cd41b34071c42a
paper_authors.name: Trong-Duong Bui
paper_authors._id: 6807ac6258cd41b34071c42b
paper_authors.name: Minh Luu Quang
paper_authors._id: 6807ac6258cd41b34071c42c
paper_authors.name: Tan-Huong Pham
paper_authors._id: 6807ac6258cd41b34071c42d
paper_authors.name: Truong-Son Hy
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-12T18:34:57.982Z
paper_authors._id: 68221c9b9f55842b8f6e0c31
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/ov4_FLLaOc5ifehP-5i0i.jpeg
paper_authors.fullname: Truong-Son Hy
paper_authors.type: user
paper_authors.user: hytruongson
paper_title: SilVar-Med: A Speech-Driven Visual Language Model for Explainable
    Abnormality Detection in Medical Imaging
paper_summary: Medical Visual Language Models have shown great potential in various
  healthcare applications, including medical image captioning and diagnostic
  assistance. However, most existing models rely on text-based instructions,
  limiting their usability in real-world clinical environments especially in
  scenarios such as surgery, text-based interaction is often impractical for
  physicians. In addition, current medical image analysis models typically lack
  comprehensive reasoning behind their predictions, which reduces their
  reliability for clinical decision-making. Given that medical diagnosis errors
  can have life-changing consequences, there is a critical need for interpretable
  and rational medical assistance. To address these challenges, we introduce an
  end-to-end speech-driven medical VLM, SilVar-Med, a multimodal medical image
  assistant that integrates speech interaction with VLMs, pioneering the task of
  voice-based communication for medical image analysis. In addition, we focus on
  the interpretation of the reasoning behind each prediction of medical
  abnormalities with a proposed reasoning dataset. Through extensive experiments,
  we demonstrate a proof-of-concept study for reasoning-driven medical image
  interpretation with end-to-end speech interaction. We believe this work will
  advance the field of medical AI by fostering more transparent, interactive, and
  clinically viable diagnostic support systems. Our code and dataset are publicly
  available at SiVar-Med.
paper_ai_summary: SilVar-Med is an end-to-end speech-driven medical visual language model that enhances interpretability and interaction in medical image analysis through voice commands and reasoning-driven predictions.
paper_ai_keywords: speech-driven medical visual language models multimodal medical image assistant voice-based communication medical image analysis reasoning-driven medical image interpretation