add a full version of the readme.d

README.md

@@ -1,3 +1,52 @@
 ---
 license: unknown
 ---
+# LLaVA1.5-BiomedCLIP-Vicuna-7b for multimodal radiology report generation
+
+This is a model based on LLaVA1.5-Vicuna-7b, finetuned to generate medical reports, based on a chest X-ray and a prompt, in our case, the instruction was "write the finding section of a chest x-ray radiology report".
+
+The vision-encoder of the model is a [BiomedCLIP](https://huggingface.co/microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224), the conector is a 2 layer MLP and the LLM is a Vicuna-7b-1.5v
+
+The dataset used for finetuning was the MIMIC-CXR shared for the challenge in Radiology Report Generation for the Association for Computational Linguistics 2024 at BioNLP Workshop.
+
+We used the 148,374 findings of MIMIC-CXR for finetuning during 3 epochs.
+
+The model metrics on the 1,063 samples of the hidden test set of the ACL challenge are the following:
+
+| Method                        | BLEU-4 | Rouge-L | Bertscore | F1-CheXbert | F1-RadGraph | Avg   |
+|-------------------------------|--------|---------|-----------|-------------|-------------|-------|
+| llava1.5-biomedclip-Vicuna-7b |  3.48  |  16.31  |   35.49   |    29.37    |    15.51    | 20.03 |
+
+
+The metrics were calculated direcly by the challenge organizer, however you can reproduce them with the following 
+example code:
+
+```python
+import json
+import logging
+from vilmedic.blocks.scorers.scores import compute_scores
+
+refs = [
+    "The lungs are clear. The cardiomediastinal silhouette is within normal limits. No acute osseous abnormalities.",
+    "The lungs are clear.There is no pleural effusion or pneumothorax.The cardiomediastinal silhouette is normal."
+]
+hyps = [
+    "The lungs are clear. There is no pleural effusion or pneumothorax. The cardiomediastinal silhouette is normal.",
+    "The lungs are clear. The cardiomediastinal silhouette is within normal limits. No acute osseous abnormalities."
+]
+print("Computing metrics, this can take a while...")
+print(json.dumps(compute_scores(["ROUGEL", "bertscore", "radgraph", "BLEU", "chexbert"],
+                                refs=refs,
+                                hyps=hyps,
+                                split=None,
+                                seed=None,
+                                config=None,
+                                epoch=None,
+                                logger=logging.getLogger(__name__),
+                                dump=False),
+                 indent=4)
+      )
+```
+
+More details of the challenge can be found on the [challenge web page](https://stanford-aimi.github.io/RRG24/)
+or in [workshop site](https://aclweb.org/aclwiki/BioNLP_Workshop)