readme: ---
  language:
  - en
  tags:
  - multimodal
  - language
  - vision
  - image-search
  - pytorch
  license:
  - mit
  metrics:
  - MRR
  ---
  
  ### Model Card: clip-imageclef
  
  ### Model Details
  
  [OpenAI CLIP model](https://openai.com/blog/clip/) fine-tuned using image-caption pairs from the [Caption Prediction dataset](https://www.imageclef.org/2017/caption) provided for the ImageCLEF 2017 competition. The model was evaluated using before and after fine-tuning, MRR@10 were 0.57 and 0.88 respectively.
  
  ### Model Date
  
  September 6, 2021
  
  ### Model Type
  
  The base model is the OpenAI CLIP model. It uses a ViT-B/32 Transformer architecture as an image encoder and uses a masked self-attention Transformer as a text encoder. These encoders are trained to maximize the similarity of (image, text) pairs via a contrastive loss.
  
  ### Fine-tuning
  
  The fine-tuning can be reproduced using code from the Github repository [elsevierlabs-os/clip-image-search]([https://github.com/elsevierlabs-os/clip-image-search#fine-tuning).
  
  ### Usage
  
  ```python
  from transformers import CLIPModel, CLIPProcessor
  
  model = CLIPModel.from_pretrained("sujitpal/clip-imageclef")
  processor = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")
  inputs = processor(text=captions, images=images, 
                     return_tensors="pt", padding=True)
  output = model(**inputs)
  ```
  
  ### Performance
  
  | Model-name                       | k=1   | k=3   | k=5   | k=10  | k=20  |
  | -------------------------------- | ----- | ----- | ----- | ----- | ----- |
  | zero-shot CLIP (baseline)        | 0.426 | 0.534 | 0.558 | 0.573 | 0.578 |
  | clip-imageclef (this model)      | 0.802 | 0.872 | 0.877 | 0.879 | 0.880 |
  
  