update README

README.md

@@ -9,7 +9,7 @@ library_name: open_clip
 pipeline_tag: zero-shot-image-classification
 ---
 
-# Model Card for WhyLesionClip
+# Model Card for WhyLesionCLIP 👍🏽
 
 #  Table of Contents
 
@@ -22,7 +22,8 @@ pipeline_tag: zero-shot-image-classification
 
 ## Model Details
 
-- **Model:** WhyLesionClip
+WhyLesionCLIP can align skin lesion images with text descriptions. It is fine-tuned from [OpenCLIP (ViT-L/14)](https://huggingface.co/laion/CLIP-ViT-L-14-laion2B-s32B-b82K) on [ISIC](https://gallery.isic-archive.com/#!/topWithHeader/onlyHeaderTop/gallery) with clinical reports generated by GPT-4V. WhyLesionCLIP significantly outperforms PubMedCLIP, BioMedCLIP, etc. in zero-shot and linear probing on various skin lesion datasets. (See results in [Evaluation](#evaluation)) While our CLIP models excel with careful data curation, training converges quickly, suggesting the current contrastive objective might not fully exploit the information from the data, potentially taking shortcuts, such as comparing images from different patients instead of focusing on diseases. Future research should explore more suitable objectives and larger-scale data collections to develop more robust medical foundation models.
+
 - **Paper:** https://arxiv.org/pdf/2405.14839
 - **Website:** https://yueyang1996.github.io/knobo/
 - **Repository:** https://github.com/YueYANG1996/KnoBo
@@ -32,84 +33,83 @@ pipeline_tag: zero-shot-image-classification
 
 Use the code below to get started with the model.
 
-[More Information Needed]
-
-
-## Uses
+```bash
+pip install open_clip_torch
+```
 
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+```python
+import torch
+from PIL import Image
+import open_clip
 
-### Direct Use
+model, _, preprocess = open_clip.create_model_and_transforms("hf-hub:yyupenn/whylesionclip")
+model.eval()
+tokenizer = open_clip.get_tokenizer("ViT-L-14")
 
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+image = preprocess(Image.open("test_skin.jpg")).unsqueeze(0)
+text = tokenizer(["dark brown", "bleeding", "irregular shape"])
 
-[More Information Needed]
+with torch.no_grad(), torch.cuda.amp.autocast():
+ image_features = model.encode_image(image)
+ text_features = model.encode_text(text)
+ image_features /= image_features.norm(dim=-1, keepdim=True)
+ text_features /= text_features.norm(dim=-1, keepdim=True)
 
-### Downstream Use
+ text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)
 
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-
-[More Information Needed]
+print("Label probs:", text_probs)
+```
 
-### Out-of-Scope Use
 
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+## Uses
 
-[More Information Needed]
+As per the original [OpenAI CLIP model card](https://github.com/openai/CLIP/blob/d50d76daa670286dd6cacf3bcd80b5e4823fc8e1/model-card.md), this model is intended as a research output for research communities. We hope that this model will enable researchers to better understand and explore zero-shot medical image (skin lesion) classification. We also hope it can be used for interdisciplinary studies of the potential impact of such models.
 
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+### Direct Use
 
+WhyLesionCLIP can be used for zero-shot skin lesion classification. You can use it to compute the similarity between an skin lesion image and a text description.
 
-## Training Details
+### Downstream Use
 
-### Training Data
+WhyLesionCLIP can be used as a feature extractor for downstream tasks. You can use it to extract features from skin lesion images and text descriptions for other downstream tasks.
 
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+### Out-of-Scope Use
 
-[More Information Needed]
+WhyLesionCLIP should not be used for clinical diagnosis or treatment. It is not intended to be used for any clinical decision-making. Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model.
 
-### Preprocessing [optional]
 
-[More Information Needed]
+## Training Details
 
+### Training Data
+We employ the [ISIC](https://gallery.isic-archive.com/#!/topWithHeader/onlyHeaderTop/gallery) dataset and use GPT-4V to generate clinical reports for 56,590 images. We preprocess these reports by extracting medically relevant findings, each described in a short and concise term. In total, we assemble 438K image-text pairs for training WhyLesionCLIP.
 
-### Training Hyperparameters
+### Training Details
 
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+We utilize the training script from [OpenCLIP](https://github.com/mlfoundations/open_clip) and select [ViT-L/14](https://huggingface.co/laion/CLIP-ViT-L-14-laion2B-s32B-b82K) as the backbone. Training is performed on 4 RTX A6000 GPUs for 10 epochs with a batch size of 128 and a learning rate of 1e−5. We choose checkpoints based on the lowest contrastive loss on validation sets.
 
 ## Evaluation
 
-<!-- This section describes the evaluation protocols and provides the results. -->
-
 ### Testing Data
 
-<!-- This should link to a Dataset Card if possible. -->
+We evaluate on 5 skin lesion classification datasets: [HAM10000](https://www.kaggle.com/datasets/kmader/skin-cancer-mnist-ham10000), [BCN20000](https://challenge.isic-archive.com/landing/2019/), [PAD-UFES-20](https://www.kaggle.com/datasets/mahdavi1202/skin-cancer), [Melanoma](https://www.kaggle.com/datasets/hasnainjaved/melanoma-skin-cancer-dataset-of-10000-images), and [UWaterloo](https://uwaterloo.ca/vision-image-processing-lab/research-demos/skin-cancer-detection). We report the zero-shot and linear probing accuracy on the above 5 datasets.
 
-[More Information Needed]
-
-
-### Metrics
-
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-
-[More Information Needed]
+### Baselines
+We compare various CLIP models, including [OpenAI-CLIP](https://huggingface.co/openai/clip-vit-large-patch14), [OpenCLIP](https://huggingface.co/laion/CLIP-ViT-L-14-laion2B-s32B-b82K), [PubMedCLIP](https://huggingface.co/flaviagiammarino/pubmed-clip-vit-base-patch32), [BioMedCLIP](https://huggingface.co/microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224), [PMC-CLIP](https://huggingface.co/ryanyip7777/pmc_vit_l_14) and [MedCLIP](https://github.com/RyanWangZf/MedCLIP). We evaluate these models in both zero-shot and linear probe scenarios. In zero-shot, GPT-4 generates prompts for each class, and we use the ensemble of cosine similarities between the image and prompts as the score for each class. In linear probing, we use the CLIP models as image encoders to extract features for logistic regression. Additionally, we include [DenseNet-121](https://arxiv.org/pdf/1608.06993) (fine-tuned on the pretraining datasets with cross-entropy loss) as a baseline for linear probing.
 
 ### Results
+The figure below shows the averaged Zero-shot and Linear Probe performance of different models on five skin lesion datasets.
 
-[More Information Needed]
-
+![Results](skin-lesion-results.png)
 
 ## Citation
 
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-
-**BibTeX:**
+Please cite our paper if you use this model in your work:
 
 ```
 @article{yang2024textbook,
-      title={A Textbook Remedy for Domain Shifts: Knowledge Priors for Medical Image Analysis}, 
-      author={Yue Yang and Mona Gandhi and Yufei Wang and Yifan Wu and Michael S. Yao and Chris Callison-Burch and James C. Gee and Mark Yatskar},
-      journal={arXiv preprint arXiv:2405.14839},
-      year={2024}
+ title={A Textbook Remedy for Domain Shifts: Knowledge Priors for Medical Image Analysis}, 
+ author={Yue Yang and Mona Gandhi and Yufei Wang and Yifan Wu and Michael S. Yao and Chris Callison-Burch and James C. Gee and Mark Yatskar},
+ journal={arXiv preprint arXiv:2405.14839},
+ year={2024}
 }
 ```

test.py

@@ -0,0 +1,20 @@
+import torch
+from PIL import Image
+import open_clip
+
+model, _, preprocess = open_clip.create_model_and_transforms("hf-hub:yyupenn/whylesionclip")
+model.eval()
+tokenizer = open_clip.get_tokenizer("ViT-L-14")
+
+image = preprocess(Image.open("test_skin.jpg")).unsqueeze(0)
+text = tokenizer(["dark brown", "bleeding", "irregular shape"])
+
+with torch.no_grad(), torch.cuda.amp.autocast():
+ image_features = model.encode_image(image)
+ text_features = model.encode_text(text)
+ image_features /= image_features.norm(dim=-1, keepdim=True)
+ text_features /= text_features.norm(dim=-1, keepdim=True)
+
+ text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)
+
+print("Label probs:", text_probs)