MedVisionLlama: Leveraging Pre-Trained Large Language Model Layers to
  Enhance Medical Image Segmentation
Large Language Models (LLMs), known for their versatility in textual data,
are increasingly being explored for their potential to enhance medical image
segmentation, a crucial task for accurate diagnostic imaging. This study
explores enhancing Vision Transformers (ViTs) for medical image segmentation by
integrating pre-trained LLM transformer blocks. Our approach, which
incorporates a frozen LLM transformer block into the encoder of a ViT-based
model, leads to substantial improvements in segmentation performance across
various medical imaging modalities. We propose a Hybrid Attention Mechanism
that combines global and local feature learning with a Multi-Scale Fusion Block
for aggregating features across different scales. The enhanced model shows
significant performance gains, including an average Dice score increase from
0.74 to 0.79 and improvements in accuracy, precision, and the Jaccard Index.
These results demonstrate the effectiveness of LLM-based transformers in
refining medical image segmentation, highlighting their potential to
significantly boost model accuracy and robustness. The source code and our
implementation are available at: https://bit.ly/3zf2CVs
66ff6a133ed87ca81d83e86a
Gurucharan Marthi Krishna Kumar
admin_assigned
2024-10-04T09:08:47.298Z
6660801ecf3bb553226c946d
/avatars/7ed4feccc3077668333f454a83101312.svg
Gurucharan Marthi Krishna Kumar
user
gurucharan-marthi
66ff6a133ed87ca81d83e86b
Aman Chadha
extracted_confirmed
2024-10-04T04:39:08.781Z
63a4754927f1f64ed7238dac
https://cdn-avatars.huggingface.co/v1/production/uploads/63a4754927f1f64ed7238dac/aH-eJF-31g4vof9jv2gmI.jpeg
Aman Chadha
user
amanchadha
66ff6a133ed87ca81d83e86c
Janine Mendola
66ff6a133ed87ca81d83e86d
Amir Shmuel
MedVisionLlama: Leveraging Pre-Trained Large Language Model Layers to
  Enhance Medical Image Segmentation
Large Language Models (LLMs), known for their versatility in textual data,
are increasingly being explored for their potential to enhance medical image
segmentation, a crucial task for accurate diagnostic imaging. This study
explores enhancing Vision Transformers (ViTs) for medical image segmentation by
integrating pre-trained LLM transformer blocks. Our approach, which
incorporates a frozen LLM transformer block into the encoder of a ViT-based
model, leads to substantial improvements in segmentation performance across
various medical imaging modalities. We propose a Hybrid Attention Mechanism
that combines global and local feature learning with a Multi-Scale Fusion Block
for aggregating features across different scales. The enhanced model shows
significant performance gains, including an average Dice score increase from
0.74 to 0.79 and improvements in accuracy, precision, and the Jaccard Index.
These results demonstrate the effectiveness of LLM-based transformers in
refining medical image segmentation, highlighting their potential to
significantly boost model accuracy and robustness. The source code and our
implementation are available at: https://bit.ly/3zf2CVs
Integrating pre-trained LLM transformer blocks into ViT encoders enhances medical image segmentation performance through a Hybrid Attention Mechanism and Multi-Scale Fusion Block, improving metrics like Dice score, accuracy, precision, and Jaccard Index.
Large Language Models Vision Transformers transformer blocks encoder Hybrid Attention Mechanism Multi-Scale Fusion Block Dice score Jaccard Index