Hugging Face's logo

X-iZhang
/
libra-Llama-3.2-3B-Instruct

Image-Text-to-Text
Transformers
Safetensors
libra
text-generation
RRG
Radiology Report Generation
Chest X-ray
Multimodal Large Language Models
conversational
Model card Files Community
libra-Llama-3.2-3B-Instruct
File size: 4,688 Bytes 
92460a9
 
 
 
81c5559
92460a9
c04030c
92460a9
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
 
---
license: apache-2.0
pipeline_tag: image-text-to-text
base_model:
- meta-llama/Llama-3.2-3B-Instruct
- microsoft/rad-dino
base_model_relation: merge
library_name: transformers
tags:
- RRG
- Radiology Report Generation
- Chest X-ray
- Multimodal Large Language Models
---
<br>

# **Libra Model Card**

**Version**: Libra-v1.0  

## Overview

**Libra** is a multimodal Large Language Model (LLM) specialized in **radiology report generation**, particularly **chest X-ray** interpretations. It can produce detailed _Findings_ sections with **temporal comparisons** (e.g., comparing a current chest X-ray with prior ones). Libra integrates the following key components:

- **RAD-DINO**: A vision encoder pre-trained on medical imaging datasets for robust feature extraction from chest X-rays.  
- **Meditron-7B**: A 7B-parameter large language model (based on Llama-2) specialized in medical text generation.  
- **Temporal Alignment Connector (TAC)**: A custom adapter that fuses features across multiple time points to enable temporal comparisons.

This model card provides an overview of Libra’s architecture, training methodology, limitations, and recommended usage guidelines.

---

##  Paper and Resources

For more detailed information regarding Libra’s methodology, theoretical foundation, and performance benchmarks, please refer to the following resources:

- **Project Website**: [Libra v1.0](https://x-izhang.github.io/Libra_v1.0/)  
- **Paper**: [arXiv:2411.19378](https://arxiv.org/abs/2411.19378)  
- **Code Repository**: [X-iZhang/Libra (GitHub)](https://github.com/X-iZhang/Libra)

Or check out our Spaces demo! [![Open in Spaces](https://huggingface.co/datasets/huggingface/badges/resolve/main/open-in-hf-spaces-md-dark.svg)](https://huggingface.co/spaces/X-iZhang/Libra)


---

##  Training Strategy

Libra is trained in a **two-stage process**:

1. **Temporal Feature Alignment**  
   - Trains TAC to effectively fuse and align features from different time points (current and previous chest X-rays).  
   - Focuses on capturing notable changes (e.g., appearance or progression of opacities, devices, and lines).

2. **Fine-Tuning for Radiology Report Generation**  
   - The language model part is fine-tuned on a large dataset of paired chest X-ray images and radiology reports.  
   - Emphasizes the generation of the _Findings_ section, especially incorporating temporal descriptors.

---
   
##  Intended Use

Libra is primarily designed to **assist** clinical practitioners, researchers, and medical students in generating chest X-ray reports. Key applications include:

- **Clinical Decision Support**: Providing draft findings that can be refined by a radiologist.  
- **Educational Tool**: Demonstrating example interpretations and temporal changes for training radiology residents.  
- **Research**: Facilitating studies on automated report generation and temporal feature learning in medical imaging.

> **Important**: Outputs should be reviewed by qualified radiologists or medical professionals before final clinical decisions are made.

---

##  Limitations and Recommendations

1. **Data Bias**: The model’s performance may be less reliable for underrepresented demographics or rare pathologies.  
2. **Clinical Oversight**: Always involve a medical professional to verify the results—Libra is not a substitute for professional judgment.  
3. **Temporal Inaccuracies**: Despite TAC’s focus on temporal alignment, subtle or uncommon changes may go unrecognized.  
4. **Generalization**: Libra’s performance on chest X-ray types or conditions not seen during training may be limited.

---

##  Ethical Considerations

- **Patient Privacy**: Ensure the data is fully de-identified and compliant with HIPAA/GDPR (or relevant privacy regulations).  
- **Responsible Use**: Deploy Libra’s outputs carefully; they are not guaranteed to be error-free.  
- **Accountability**: Users and organizations must assume responsibility for verifying clinical accuracy and safety.

---

## How to Cite ✒️

If you use Libra in academic or research contexts, please cite:

```bibtex
@misc{zhang2024libraleveragingtemporalimages,
      title={Libra: Leveraging Temporal Images for Biomedical Radiology Analysis}, 
      author={Xi Zhang and Zaiqiao Meng and Jake Lever and Edmond S. L. Ho},
      year={2024},
      eprint={2411.19378},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2411.19378}, 
}
```
##  Disclaimer:
This tool is for research and educational purposes only. It is not FDA-approved or CE-marked for clinical use. Users should consult qualified healthcare professionals for any clinical decisions.