Upload 8 files

README.md

@@ -0,0 +1,137 @@
+---
+library_name: transformers
+pipeline_tag: image-text-to-text
+license: apache-2.0
+datasets:
+- joshuachou/SkinCAP
+- HemanthKumarK/SKINgpt
+language:
+- en
+tags:
+- biology
+- skin
+- skin disease
+- cancer
+- medical
+---
+# Model Card for PaliGemma Dermatology Model
+
+## Model Details
+
+### Model Description
+
+This model, based on the PaliGemma-3B architecture, has been fine-tuned for dermatology-related image and text processing tasks. The model is designed to assist in the identification of various skin conditions using a combination of image analysis and natural language processing.
+
+
+- **Developed by:** Bruce_Wayne
+- **Model type:** vision model
+- **Finetuned from model:** https://huggingface.co/google/paligemma-3b-pt-224
+- **LoRa Adaptors used:** Yes
+- **Intended use:** Medical image analysis, specifically for dermatology
+**
+### please let me know how the model works -->https://forms.gle/cBA6apSevTyiEbp46
+### Thank you
+## Uses
+### Direct Use
+
+The model can be directly used for analyzing dermatology images, providing insights into potential skin conditions.
+
+
+## Bias, Risks, and Limitations
+
+**Skin Tone Bias:** The model may have been trained on a dataset that does not adequately represent all skin tones, potentially leading to biased results.
+**Geographic Bias:** The model's performance may vary depending on the prevalence of certain conditions in different geographic regions.
+
+## How to Get Started with the Model
+
+```python
+
+import torch
+from transformers import AutoProcessor, PaliGemmaForConditionalGeneration
+from PIL import Image
+
+# Load the model and processor
+model_id = "brucewayne0459/paligemma_derm"
+processor = AutoProcessor.from_pretrained(model_id)
+model = PaliGemmaForConditionalGeneration.from_pretrained(model_id, device_map={"": 0})
+model.eval()
+
+# Load a sample image and text input
+input_text = "Identify the skin condition?"
+input_image_path = " Replace with your actual image path"  
+input_image = Image.open(input_image_path).convert("RGB")
+
+# Process the input
+inputs = processor(text=input_text, images=input_image, return_tensors="pt", padding="longest").to("cuda" if torch.cuda.is_available() else "cpu")
+
+# Set the maximum length for generation
+max_new_tokens = 50
+
+# Run inference
+with torch.no_grad():
+    outputs = model.generate(**inputs, max_new_tokens=max_new_tokens)
+
+# Decode the output
+decoded_output = processor.decode(outputs[0], skip_special_tokens=True)
+print("Model Output:", decoded_output)
+```
+## Training Details
+
+### Training Data
+
+The model was fine-tuned on a dataset of dermatological images combined with disease names
+
+### Training Procedure
+
+The model was fine-tuned using LoRA (Low-Rank Adaptation) for more efficient training. Mixed precision (bfloat16) was used to speed up training and reduce memory usage.
+
+#### Training Hyperparameters
+
+- **Training regime:** Mixed precision (bfloat16)
+- **Epochs:** 10
+- **Learning rate:** 2e-5
+- **Batch size:** 6
+- **Gradient accumulation steps:** 4
+
+
+## Evaluation
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+The model was evaluated on a separate validation set of dermatological images and Disease Names, distinct from the training data.
+
+#### Metrics
+- **Validation Loss:** The loss was tracked throughout the training process to evaluate model performance.
+- **Accuracy:** The primary metric for assessing model predictions.
+### Results
+
+The model achieved a final validation loss of approximately 0.2214, indicating reasonable performance in predicting skin conditions based on the dataset used.
+
+#### Summary
+
+
+## Environmental Impact
+
+
+- **Hardware Type:** 1 x L4 GPU
+- **Hours used:** ~22 HOURS
+- **Cloud Provider:** LIGHTNING AI
+- **Compute Region:** USA
+- **Carbon Emitted:** 0.9 kg eq. CO2
+
+## Technical Specifications
+
+### Model Architecture and Objective
+
+- **Architecture:** Vision-Language model based on PaliGemma-3B
+- **Objective:** To classify and diagnose dermatological conditions from images and text
+
+### Compute Infrastructure
+
+#### Hardware
+
+- **GPU:** 1xL4 GPU
+## Model Card Authors 
+Bruce_Wayne

config.json

@@ -0,0 +1,38 @@
+{
+  "_name_or_path": "google/paligemma-3b-pt-224",
+  "architectures": [
+    "PaliGemmaForConditionalGeneration"
+  ],
+  "bos_token_id": 2,
+  "eos_token_id": 1,
+  "hidden_size": 2048,
+  "image_token_index": 257152,
+  "model_type": "paligemma",
+  "pad_token_id": 0,
+  "projection_dim": 2048,
+  "text_config": {
+    "hidden_size": 2048,
+    "intermediate_size": 16384,
+    "model_type": "gemma",
+    "num_attention_heads": 8,
+    "num_hidden_layers": 18,
+    "num_image_tokens": 256,
+    "num_key_value_heads": 1,
+    "torch_dtype": "float32",
+    "vocab_size": 257216
+  },
+  "torch_dtype": "float32",
+  "transformers_version": "4.45.0.dev0",
+  "vision_config": {
+    "hidden_size": 1152,
+    "intermediate_size": 4304,
+    "model_type": "siglip_vision_model",
+    "num_attention_heads": 16,
+    "num_hidden_layers": 27,
+    "num_image_tokens": 256,
+    "patch_size": 14,
+    "projection_dim": 2048,
+    "projector_hidden_act": "gelu_fast",
+    "vision_use_head": false
+  }
+}

generation_config.json

@@ -0,0 +1,7 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 2,
+  "eos_token_id": 1,
+  "pad_token_id": 0,
+  "transformers_version": "4.45.0.dev0"
+}

gitattributes

@@ -0,0 +1,36 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

model-00002-of-00003.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5cfdf65953f036f1789d3f9136758620442f1a64de31372b90204466578b8c16
+size 4999820608

preprocessor_config.json

@@ -0,0 +1,25 @@
+{
+  "do_convert_rgb": null,
+  "do_normalize": true,
+  "do_rescale": true,
+  "do_resize": true,
+  "image_mean": [
+    0.5,
+    0.5,
+    0.5
+  ],
+  "image_processor_type": "SiglipImageProcessor",
+  "image_seq_length": 256,
+  "image_std": [
+    0.5,
+    0.5,
+    0.5
+  ],
+  "processor_class": "PaliGemmaProcessor",
+  "resample": 3,
+  "rescale_factor": 0.00392156862745098,
+  "size": {
+    "height": 224,
+    "width": 224
+  }
+}

special_tokens_map.json

@@ -0,0 +1,39 @@
+{
+  "additional_special_tokens": [
+    {
+      "content": "<image>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    }
+  ],
+  "bos_token": {
+    "content": "<bos>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<eos>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<pad>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}