title: Enhancing Abnormality Grounding for Vision Language Models with
    Knowledge Descriptions
summary: Visual Language Models (VLMs) have demonstrated impressive capabilities in
  visual grounding tasks. However, their effectiveness in the medical domain,
  particularly for abnormality detection and localization within medical images,
  remains underexplored. A major challenge is the complex and abstract nature of
  medical terminology, which makes it difficult to directly associate
  pathological anomaly terms with their corresponding visual features. In this
  work, we introduce a novel approach to enhance VLM performance in medical
  abnormality detection and localization by leveraging decomposed medical
  knowledge. Instead of directly prompting models to recognize specific
  abnormalities, we focus on breaking down medical concepts into fundamental
  attributes and common visual patterns. This strategy promotes a stronger
  alignment between textual descriptions and visual features, improving both the
  recognition and localization of abnormalities in medical images.We evaluate our
  method on the 0.23B Florence-2 base model and demonstrate that it achieves
  comparable performance in abnormality grounding to significantly larger 7B
  LLaVA-based medical VLMs, despite being trained on only 1.5% of the data used
  for such models. Experimental results also demonstrate the effectiveness of our
  approach in both known and previously unseen abnormalities, suggesting its
  strong generalization capabilities.
paper_authors._id: 67c94e5f8c4ef8be73583f4b
paper_authors.name: Jun Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-09T15:59:20.528Z
paper_authors._id: 6526523fa754153bd4da45f3
paper_authors.avatarUrl: /avatars/848b234fcd59d9885ddb673ffea20a63.svg
paper_authors.fullname: RioJune
paper_authors.type: user
paper_authors.user: RioJune
paper_authors._id: 67c94e5f8c4ef8be73583f4c
paper_authors.name: Che Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-06T10:03:06.882Z
paper_authors._id: 631b9ff5824f2502e3557c7e
paper_authors.avatarUrl: /avatars/076043c9dba07644a570692563ef8114.svg
paper_authors.fullname: liu
paper_authors.type: user
paper_authors.user: che111
paper_authors._id: 67c94e5f8c4ef8be73583f4d
paper_authors.name: Wenjia Bai
paper_authors._id: 67c94e5f8c4ef8be73583f4e
paper_authors.name: Rossella Arcucci
paper_authors._id: 67c94e5f8c4ef8be73583f4f
paper_authors.name: Cosmin I. Bercea
paper_authors._id: 67c94e5f8c4ef8be73583f50
paper_authors.name: Julia A. Schnabel
paper_title: Enhancing Abnormality Grounding for Vision Language Models with
    Knowledge Descriptions
paper_summary: Visual Language Models (VLMs) have demonstrated impressive capabilities in
  visual grounding tasks. However, their effectiveness in the medical domain,
  particularly for abnormality detection and localization within medical images,
  remains underexplored. A major challenge is the complex and abstract nature of
  medical terminology, which makes it difficult to directly associate
  pathological anomaly terms with their corresponding visual features. In this
  work, we introduce a novel approach to enhance VLM performance in medical
  abnormality detection and localization by leveraging decomposed medical
  knowledge. Instead of directly prompting models to recognize specific
  abnormalities, we focus on breaking down medical concepts into fundamental
  attributes and common visual patterns. This strategy promotes a stronger
  alignment between textual descriptions and visual features, improving both the
  recognition and localization of abnormalities in medical images.We evaluate our
  method on the 0.23B Florence-2 base model and demonstrate that it achieves
  comparable performance in abnormality grounding to significantly larger 7B
  LLaVA-based medical VLMs, despite being trained on only 1.5% of the data used
  for such models. Experimental results also demonstrate the effectiveness of our
  approach in both known and previously unseen abnormalities, suggesting its
  strong generalization capabilities.
paper_ai_summary: A new approach decomposes medical knowledge into fundamental attributes and visual patterns to enhance visual language models in medical abnormality detection and localization, achieving comparable performance to larger models with less data.
paper_ai_keywords: Visual Language Models VLMs abnormality detection localization medical images decomposed medical knowledge textual descriptions visual features Florence-2 base model LLaVA-based models