README.md

metadata

tags:
  - sentence-transformers
  - sentence-similarity
  - feature-extraction
  - dense
  - generated_from_trainer
  - dataset_size:105
  - loss:MultipleNegativesRankingLoss
base_model: sentence-transformers/clip-ViT-B-32
widget:
  - source_sentence: >-
      The image depicts multiple small, soft, flesh-colored to slightly
      hyperpigmented skin lesions located on the eyelid and surrounding skin.
      These lesions are pedunculated with a smooth surface, typical of skin tags
      (acrochordons). Some are more prominent and raised, while others are
      flatter against the skin surface. The surrounding skin appears slightly
      wrinkled, consistent with normal aging changes.
    sentences:
      - >-
        The image shows a reticulated, net-like pattern of erythema and
        hyperpigmentation on the lower back. The lesions appear to be
        brownish-red in color with defined, irregular borders. There is no
        evidence of blistering or active inflammation. The distribution is
        linear and symmetric, suggesting exposure to a patterned heat source.
      - >-
        The image shows a right lower leg with marked erythema covering most of
        the calf and extending around the ankle. The redness has an ill-defined
        border and there is noticeable swelling. The skin appears smooth and
        shiny, suggesting edema. There is no evidence of open wounds or pus. The
        distribution of redness and swelling is primarily around the shin and
        calf areas.
      - >-
        The image displays toenails with signs of fungal infection. The nails
        appear thickened and discolored, predominantly yellowish-brown. There is
        noticeable onycholysis, especially in the larger toenail, with crumbling
        and irregular borders. The surface of the nails looks rough and uneven,
        and there is subungual debris. The surrounding skin appears intact
        without visible erythema or scaling.
  - source_sentence: >-
      The image shows a circular, red lesion on the skin, located on the cheek.
      It features a central elevated red spot with radiating capillaries,
      resembling a spider's web. The surrounding skin appears slightly
      erythematous. The lesion has well-defined borders and is isolated with no
      other lesions visible in the immediate vicinity.
    sentences:
      - >-
        The image shows the lower leg and ankle area with diffuse erythema,
        swelling, and warmth. The skin appears red, indicating inflammation, and
        there is a shiny quality suggesting edema. The borders of the affected
        area are poorly defined, blending into the surrounding skin. No visible
        pus or open wounds are present. The distribution is localized to the
        leg, suggesting a likely case of cellulitis.
      - >-
        The image shows multiple skin-colored to slightly hyperpigmented, soft,
        pedunculated papules consistent with skin tags. These lesions are
        located on the skin, likely in a non-exposed area of the body. They
        appear to have smooth surfaces, with some variation in size ranging from
        a few millimeters to larger. The surrounding skin is unremarkable,
        showing no signs of erythema or scaling. There are also some fine hairs
        visible, and no signs of irritation or secondary changes are noted
        around the lesions.
      - >-
        The image shows an area of skin with multiple erythematous, excoriated
        papules and plaques, some of which are coalescing. The lesions are
        distributed on the flexural aspect of the extremity. The skin appears
        dry and lichenified with some areas showing slight scaling. Borders of
        the individual lesions are irregular, and there is no clear demarcation
        from the surrounding skin.
  - source_sentence: >-
      The image shows an ulcerative lesion on the inner mucosal surface of the
      lower lip. The ulcer is round, shallow, and well-demarcated with a white
      to yellowish base and an erythematous halo. The surrounding mucosa appears
      slightly swollen. The lesion is singular, with no apparent vesicles or
      additional ulcers visible in this image.
    sentences:
      - >-
        The image shows the dorsal aspect of a hand with multiple erythematous,
        scaly patches and plaques. These lesions are well-demarcated with
        lichenification indicating chronicity. The plaques exhibit excoriations,
        suggesting pruritus. The skin appears dry and rough, primarily affecting
        the knuckles and the back of the hand.
      - >-
        The image shows erythematous patches and papules located on the cheek
        near the mouth. The lesions exhibit poorly defined borders and are
        accompanied by visible dryness and mild scaling. The skin appears
        slightly thickened and there is evidence of mild lichenification,
        suggesting chronicity. The overall distribution is localized on the face
        with no visible excoriations or crusting.
      - >-
        The image shows erythematous, scaly patches located in the beard area of
        the face. The lesions exhibit a well-demarcated border and slight
        follicular involvement, with perifollicular pustules and mild scaling.
        The affected area demonstrates inflamed patches with some papules and an
        uneven surface texture, indicating a possible fungal infection.
  - source_sentence: >-
      The image shows a dermatofibroma located on the skin. It presents as a
      solitary, elevated, firm nodule with a well-defined border. The lesion is
      reddish-brown in color with a central lighter area. There are several
      hairs emerging from the surface, indicating it is on a hair-bearing area
      of the skin. The surface texture appears smooth with no visible scaling or
      ulceration.
    sentences:
      - >-
        Clusters of small, grouped vesicles on an erythematous base are present
        on the upper lip. The vesicles are translucent with clear fluid and some
        central umbilication. The surrounding skin shows mild erythema. There
        are no crusts or secondary changes visible.
      - >-
        The image shows a single, flesh-colored papule that is soft to the
        touch. It has a pedunculated shape, protruding from the surface of the
        skin. The lesion is located on an area of skin with normal texture and
        pigmentation. The borders are well-defined and the surface appears
        smooth without any signs of inflammation or scaling.
      - >-
        The image shows an irregular, serpiginous, erythematous track on the
        skin, indicative of a cutaneous larva migrans infection. The lesion
        appears slightly raised and is located on the forearm. The borders of
        the track are well-defined but uneven due to its winding nature. There
        are areas with mild inflammation along the path and no visible signs of
        secondary infection such as pustules or crusting.
  - source_sentence: >-
      The image depicts the abdominal region with a reticulated, or net-like,
      erythematous pattern on the skin. The area shows a faint, reddish-brown
      discoloration with a lacy appearance, indicative of erythema ab igne. The
      borders of the discoloration are irregular but well-demarcated. No
      blistering or open lesions are visible, and the skin surface appears
      intact without signs of scaling or crusting.
    sentences:
      - >-
        The image shows an elderly person with erythematous patches and papules
        prominently located on the central face, including the cheeks, nose, and
        forehead. The skin findings display a symmetrical distribution. The nose
        is slightly bulbous, suggesting possible rhinophyma changes. The lesions
        do not have well-defined borders and there is no visible scaling or
        pustular formations. The background skin appears sun-damaged with fine
        wrinkling, but without significant telangiectasia visible in this image.
      - >-
        The image shows a round, red, and scaly patch on the skin. The lesion
        has a well-defined, elevated border with a clear center, creating a
        ring-like appearance. The border is erythematous and slightly raised,
        while the center appears less inflamed and more skin-colored. The
        surface of the lesion is rough, indicating scaling. No secondary changes
        like crusting or blistering are visible.
      - >-
        The image depicts multiple small, soft, flesh-colored to slightly
        hyperpigmented skin lesions located on the eyelid and surrounding skin.
        These lesions are pedunculated with a smooth surface, typical of skin
        tags (acrochordons). Some are more prominent and raised, while others
        are flatter against the skin surface. The surrounding skin appears
        slightly wrinkled, consistent with normal aging changes.
datasets:
  - TatvaRA/skin-positive-negative-pairs
pipeline_tag: sentence-similarity
library_name: sentence-transformers
metrics:
  - cosine_accuracy
model-index:
  - name: SentenceTransformer based on sentence-transformers/clip-ViT-B-32
    results:
      - task:
          type: triplet
          name: Triplet
        dataset:
          name: skin dataset train
          type: skin-dataset-train
        metrics:
          - type: cosine_accuracy
            value: 1
            name: Cosine Accuracy
      - task:
          type: triplet
          name: Triplet
        dataset:
          name: skin dataset valid
          type: skin-dataset-valid
        metrics:
          - type: cosine_accuracy
            value: 1
            name: Cosine Accuracy

SentenceTransformer based on sentence-transformers/clip-ViT-B-32

This is a sentence-transformers model finetuned from sentence-transformers/clip-ViT-B-32 on the skin-positive-negative-pairs dataset. It maps sentences & paragraphs to a None-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: sentence-transformers/clip-ViT-B-32
• Maximum Sequence Length: 77 tokens
• Output Dimensionality: None dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • skin-positive-negative-pairs

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): CLIPModel()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("TatvaRA/clip-skin-embeddings")
# Run inference
sentences = [
    'The image depicts the abdominal region with a reticulated, or net-like, erythematous pattern on the skin. The area shows a faint, reddish-brown discoloration with a lacy appearance, indicative of erythema ab igne. The borders of the discoloration are irregular but well-demarcated. No blistering or open lesions are visible, and the skin surface appears intact without signs of scaling or crusting.',
    'The image depicts multiple small, soft, flesh-colored to slightly hyperpigmented skin lesions located on the eyelid and surrounding skin. These lesions are pedunculated with a smooth surface, typical of skin tags (acrochordons). Some are more prominent and raised, while others are flatter against the skin surface. The surrounding skin appears slightly wrinkled, consistent with normal aging changes.',
    'The image shows an elderly person with erythematous patches and papules prominently located on the central face, including the cheeks, nose, and forehead. The skin findings display a symmetrical distribution. The nose is slightly bulbous, suggesting possible rhinophyma changes. The lesions do not have well-defined borders and there is no visible scaling or pustular formations. The background skin appears sun-damaged with fine wrinkling, but without significant telangiectasia visible in this image.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.3884, 0.3253],
#         [0.3884, 1.0000, 0.5531],
#         [0.3253, 0.5531, 1.0000]])

Evaluation

Metrics

Triplet

• Datasets: skin-dataset-train and skin-dataset-valid
• Evaluated with TripletEvaluator

Metric	skin-dataset-train	skin-dataset-valid
cosine_accuracy	1.0	1.0

Training Details

Training Dataset

skin-positive-negative-pairs

• Dataset: skin-positive-negative-pairs at a6e13e3
• Size: 105 training samples
• Columns: anchor, positive, and negative
• Approximate statistics based on the first 105 samples:

  	anchor	positive	negative
  type	PIL.PngImagePlugin.PngImageFile	string	string
  details		min: 46 tokens mean: 75.37 tokens max: 77 tokens	min: 46 tokens mean: 75.55 tokens max: 77 tokens

• Samples:

  anchor	positive	negative
  	The image depicts a melanocytic nevus located on the skin. It is characterized by a well-defined, round brown lesion centrally located on a lighter skin background. The pigmentation appears uniform with a slightly darker speckled pattern distributed evenly throughout the nevus. The borders are smooth and clearly demarcated from the surrounding skin. There is no apparent erythema, ulceration, or significant secondary changes. Sparse hair is visible around the lesion, and the overall lesion is flat, without palpable elevation.	The image shows the upper lip, featuring a cluster of small, grouped vesicles on an erythematous base. The vesicles are clear, filled with fluid, and some are beginning to coalesce. There is minimal surrounding erythema, and the lesions are located at the vermillion border of the lip. The borders of the vesicles are well-defined, and there are no signs of crusting at this stage.
  	The image shows a well-defined, erythematous plaque on the skin with a rough, scaly surface. The lesion is located on a hand and exhibits significant inflammation and lichenification. There is evidence of excoriation and possible oozing, suggesting chronic irritation and scratching. The borders of the lesion are distinct, and the surrounding skin appears dry.	The image displays multiple open comedones, appearing as black or dark brown plugs, located on the periocular region of the face. These lesions have a characteristic distribution, clustering under the eye and on the upper cheek area. The surrounding skin exhibits signs of chronic sun damage, including fine wrinkles and a mildly rough texture. The borders of the comedones are well-defined, and there is no evidence of erythema or secondary infection around the lesions.
  	The image shows a close-up view of the skin with an irregularly shaped lesion displaying a mixture of colors, including pink, brown, and some erythematous areas. The lesion has uneven borders and appears slightly raised with scaly, crusted surfaces. There are areas of keratinization, with the lesion being distributed in a somewhat asymmetrical pattern. Surrounding skin appears less affected, with some visible telangiectasia.	The image shows the back of an individual with multiple vesicular lesions. These lesions are scattered across the skin surface and vary in size. They appear as small, red, blister-like eruptions that are slightly raised. Some lesions exhibit a clear vesicular head, while others are crusted over, indicating different stages of development. The distribution is widespread across the back, with a noticeable absence of clustering. The lesions have well-defined borders, and the surrounding skin appears otherwise normal without significant erythema or swelling.

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Evaluation Dataset

skin-positive-negative-pairs

• Dataset: skin-positive-negative-pairs at a6e13e3
• Size: 22 evaluation samples
• Columns: anchor, positive, and negative
• Approximate statistics based on the first 22 samples:

  	anchor	positive	negative
  type	PIL.PngImagePlugin.PngImageFile	string	string
  details		min: 74 tokens mean: 76.59 tokens max: 77 tokens	min: 60 tokens mean: 75.59 tokens max: 77 tokens

• Samples:

  anchor	positive	negative
  	The image shows an area of skin with multiple open comedones, commonly known as blackheads, scattered across the affected area. These comedones are predominantly located on the cheek and temple of the face. The skin appears weathered and sallow, with some areas exhibiting signs of solar damage, including wrinkling and yellowish discoloration. The borders of the lesions are irregular, and there are multiple large, open comedones. There are secondary changes, such as mild erythema and induration in some areas, suggesting chronic sun exposure.	The image displays toenails with signs of fungal infection. The nails appear thickened and discolored, predominantly yellowish-brown. There is noticeable onycholysis, especially in the larger toenail, with crumbling and irregular borders. The surface of the nails looks rough and uneven, and there is subungual debris. The surrounding skin appears intact without visible erythema or scaling.
  	The image shows a circular, red lesion on the skin, located on the cheek. It features a central elevated red spot with radiating capillaries, resembling a spider's web. The surrounding skin appears slightly erythematous. The lesion has well-defined borders and is isolated with no other lesions visible in the immediate vicinity.	The image shows an area of skin with multiple erythematous, excoriated papules and plaques, some of which are coalescing. The lesions are distributed on the flexural aspect of the extremity. The skin appears dry and lichenified with some areas showing slight scaling. Borders of the individual lesions are irregular, and there is no clear demarcation from the surrounding skin.
  	The image shows an elderly person with erythematous patches and papules prominently located on the central face, including the cheeks, nose, and forehead. The skin findings display a symmetrical distribution. The nose is slightly bulbous, suggesting possible rhinophyma changes. The lesions do not have well-defined borders and there is no visible scaling or pustular formations. The background skin appears sun-damaged with fine wrinkling, but without significant telangiectasia visible in this image.	The image shows a round, erythematous (red) lesion with a well-defined, slightly raised border. The center of the lesion appears to be clearer with minor scaling, typical of a ringworm (tinea corporis) infection. The lesion is located on skin likely from a limb or torso. The margins are distinct with some subtle scaling toward the periphery.

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: epoch
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• learning_rate: 0.0001
• num_train_epochs: 7

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: epoch
• prediction_loss_only: True
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 0.0001
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 7
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.0
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss	Validation Loss	skin-dataset-train_cosine_accuracy	skin-dataset-valid_cosine_accuracy
-1	-1	-	-	0.9619	1.0
0.1429	1	2.7596	-	-	-
0.2857	2	2.5414	-	-	-
0.4286	3	2.6204	-	-	-
0.5714	4	2.4667	-	-	-
0.7143	5	2.4014	-	-	-
0.8571	6	2.5579	-	-	-
1.0	7	2.0899	2.2113	-	-
1.1429	8	2.1459	-	-	-
1.2857	9	2.0529	-	-	-
1.4286	10	1.8848	-	-	-
1.5714	11	2.0387	-	-	-
1.7143	12	2.0944	-	-	-
1.8571	13	2.1068	-	-	-
2.0	14	1.6342	2.1221	-	-
0.1429	1	1.7895	-	-	-
0.2857	2	1.6297	-	-	-
0.4286	3	1.9204	-	-	-
0.5714	4	1.9449	-	-	-
0.7143	5	1.8608	-	-	-
0.8571	6	2.186	-	-	-
1.0	7	1.522	1.8561	-	-
1.1429	8	1.5326	-	-	-
1.2857	9	1.3086	-	-	-
1.4286	10	1.1582	-	-	-
1.5714	11	1.4275	-	-	-
1.7143	12	1.3807	-	-	-
1.8571	13	1.4184	-	-	-
2.0	14	0.909	1.7285	-	-
2.1429	15	1.2727	-	-	-
2.2857	16	1.3549	-	-	-
2.4286	17	1.0448	-	-	-
2.5714	18	1.0773	-	-	-
2.7143	19	1.1036	-	-	-
2.8571	20	0.9444	-	-	-
3.0	21	0.5148	1.6311	-	-
3.1429	22	0.8764	-	-	-
3.2857	23	0.809	-	-	-
3.4286	24	0.8724	-	-	-
3.5714	25	0.8276	-	-	-
3.7143	26	0.8923	-	-	-
3.8571	27	0.797	-	-	-
4.0	28	0.5056	1.5989	-	-
4.1429	29	0.7625	-	-	-
4.2857	30	0.6356	-	-	-
4.4286	31	0.8352	-	-	-
4.5714	32	0.6936	-	-	-
4.7143	33	0.6338	-	-	-
4.8571	34	0.5392	-	-	-
5.0	35	0.3826	1.5942	-	-
5.1429	36	0.6252	-	-	-
5.2857	37	0.5969	-	-	-
5.4286	38	0.4948	-	-	-
5.5714	39	0.7617	-	-	-
5.7143	40	0.6775	-	-	-
5.8571	41	0.6289	-	-	-
6.0	42	0.2195	1.5698	-	-
6.1429	43	0.5786	-	-	-
6.2857	44	0.6049	-	-	-
6.4286	45	0.5162	-	-	-
6.5714	46	0.5841	-	-	-
6.7143	47	0.6579	-	-	-
6.8571	48	0.6702	-	-	-
7.0	49	0.2975	1.5598	-	-
-1	-1	-	-	1.0	1.0

Framework Versions

• Python: 3.10.12
• Sentence Transformers: 5.0.0
• Transformers: 4.53.2
• PyTorch: 2.7.1+cu126
• Accelerate: 1.9.0
• Datasets: 4.0.0
• Tokenizers: 0.21.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}