title: MedVisionLlama: Leveraging Pre-Trained Large Language Model Layers to
    Enhance Medical Image Segmentation
summary: Large Language Models (LLMs), known for their versatility in textual data,
  are increasingly being explored for their potential to enhance medical image
  segmentation, a crucial task for accurate diagnostic imaging. This study
  explores enhancing Vision Transformers (ViTs) for medical image segmentation by
  integrating pre-trained LLM transformer blocks. Our approach, which
  incorporates a frozen LLM transformer block into the encoder of a ViT-based
  model, leads to substantial improvements in segmentation performance across
  various medical imaging modalities. We propose a Hybrid Attention Mechanism
  that combines global and local feature learning with a Multi-Scale Fusion Block
  for aggregating features across different scales. The enhanced model shows
  significant performance gains, including an average Dice score increase from
  0.74 to 0.79 and improvements in accuracy, precision, and the Jaccard Index.
  These results demonstrate the effectiveness of LLM-based transformers in
  refining medical image segmentation, highlighting their potential to
  significantly boost model accuracy and robustness. The source code and our
  implementation are available at: https://bit.ly/3zf2CVs
paper_authors._id: 66ff6a133ed87ca81d83e86a
paper_authors.name: Gurucharan Marthi Krishna Kumar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-04T09:08:47.298Z
paper_authors._id: 6660801ecf3bb553226c946d
paper_authors.avatarUrl: /avatars/7ed4feccc3077668333f454a83101312.svg
paper_authors.fullname: Gurucharan Marthi Krishna Kumar
paper_authors.type: user
paper_authors.user: gurucharan-marthi
paper_authors._id: 66ff6a133ed87ca81d83e86b
paper_authors.name: Aman Chadha
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-10-04T04:39:08.781Z
paper_authors._id: 63a4754927f1f64ed7238dac
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63a4754927f1f64ed7238dac/aH-eJF-31g4vof9jv2gmI.jpeg
paper_authors.fullname: Aman Chadha
paper_authors.type: user
paper_authors.user: amanchadha
paper_authors._id: 66ff6a133ed87ca81d83e86c
paper_authors.name: Janine Mendola
paper_authors._id: 66ff6a133ed87ca81d83e86d
paper_authors.name: Amir Shmuel
paper_title: MedVisionLlama: Leveraging Pre-Trained Large Language Model Layers to
    Enhance Medical Image Segmentation
paper_summary: Large Language Models (LLMs), known for their versatility in textual data,
  are increasingly being explored for their potential to enhance medical image
  segmentation, a crucial task for accurate diagnostic imaging. This study
  explores enhancing Vision Transformers (ViTs) for medical image segmentation by
  integrating pre-trained LLM transformer blocks. Our approach, which
  incorporates a frozen LLM transformer block into the encoder of a ViT-based
  model, leads to substantial improvements in segmentation performance across
  various medical imaging modalities. We propose a Hybrid Attention Mechanism
  that combines global and local feature learning with a Multi-Scale Fusion Block
  for aggregating features across different scales. The enhanced model shows
  significant performance gains, including an average Dice score increase from
  0.74 to 0.79 and improvements in accuracy, precision, and the Jaccard Index.
  These results demonstrate the effectiveness of LLM-based transformers in
  refining medical image segmentation, highlighting their potential to
  significantly boost model accuracy and robustness. The source code and our
  implementation are available at: https://bit.ly/3zf2CVs
paper_ai_keywords: Vision Transformers (ViTs) pre-trained LLM transformer blocks Hybrid Attention Mechanism global feature learning local feature learning Multi-Scale Fusion Block Dice score Jaccard Index