readme: ---
  license: apache-2.0
  ---
  
  # Chinese-CLIP-RN50
  
  ## Introduction
  This is the smallest model of the Chinese CLIP series, with ResNet-50 as the image encoder and RBT3 as the text encoder. Chinese CLIP is a simple implementation of CLIP on a large-scale dataset of around 200 million Chinese image-text pairs. For more details, please refer to our technical report https://arxiv.org/abs/2211.01335 and our official github repo https://github.com/OFA-Sys/Chinese-CLIP
  
  ## Use with the official API
  We provide a simple code snippet to show how to use the API for Chinese-CLIP. For starters, please install cn_clip:
  ```bash
  # to install the latest stable release
  pip install cn_clip
  
  # or install from source code
  cd Chinese-CLIP
  pip install -e .
  ```
  After installation, use Chinese CLIP as shown below:
  ```python
  import torch
  from PIL import Image
  
  import cn_clip.clip as clip
  from cn_clip.clip import load_from_name, available_models
  print("Available models:", available_models())  
  # Available models: ['ViT-B-16', 'ViT-L-14', 'ViT-L-14-336', 'ViT-H-14', 'RN50']
  
  device = "cuda" if torch.cuda.is_available() else "cpu"
  model, preprocess = load_from_name("RN50", device=device, download_root='./')
  model.eval()
  image = preprocess(Image.open("examples/pokemon.jpeg")).unsqueeze(0).to(device)
  text = clip.tokenize(["杰尼龟", "妙蛙种子", "小火龙", "皮卡丘"]).to(device)
  
  with torch.no_grad():
      image_features = model.encode_image(image)
      text_features = model.encode_text(text)
      # Normalize the features. Please use the normalized features for downstream tasks.
      image_features /= image_features.norm(dim=-1, keepdim=True) 
      text_features /= text_features.norm(dim=-1, keepdim=True)      
  
      logits_per_image, logits_per_text = model.get_similarity(image, text)
      probs = logits_per_image.softmax(dim=-1).cpu().numpy()
  
  print("Label probs:", probs)  # [[1.268734e-03 5.436878e-02 6.795761e-04 9.436829e-01]]
  ```
  
  However, if you are not satisfied with only using the API, feel free to check our github repo https://github.com/OFA-Sys/Chinese-CLIP for more details about training and inference. 
  <br><br>
  
  ## Results
  **MUGE Text-to-Image Retrieval**:
  <table border="1" width="100%">
      <tr align="center">
          <th>Setup</th><th colspan="4">Zero-shot</th><th colspan="4">Finetune</th>
      </tr>
      <tr align="center">
          <td>Metric</td><td>R@1</td><td>R@5</td><td>R@10</td><td>MR</td><td>R@1</td><td>R@5</td><td>R@10</td><td>MR</td>
      </tr>
  	<tr align="center">
          <td width="120%">Wukong</td><td>42.7</td><td>69.0</td><td>78.0</td><td>63.2</td><td>52.7</td><td>77.9</td><td>85.6</td><td>72.1</td>
      </tr>
  	<tr align="center">
          <td width="120%">R2D2</td><td>49.5</td><td>75.7</td><td>83.2</td><td>69.5</td><td>60.1</td><td>82.9</td><td>89.4</td><td>77.5</td>
      </tr>
  	<tr align="center">
          <td width="120%">CN-CLIP</td><td>63.0</td><td>84.1</td><td>89.2</td><td>78.8</td><td>68.9</td><td>88.7</td><td>93.1</td><td>83.6</td>
      </tr>
  </table>
  <br>
  
  **Flickr30K-CN Retrieval**:
  <table border="1" width="120%">
  	<tr align="center">
          <th>Task</th><th colspan="6">Text-to-Image</th><th colspan="6">Image-to-Text</th>
      </tr>
      <tr align="center">
          <th>Setup</th><th colspan="3">Zero-shot</th><th colspan="3">Finetune</th><th colspan="3">Zero-shot</th><th colspan="3">Finetune</th>
      </tr>
      <tr align="center">
          <td>Metric</td><td>R@1</td><td>R@5</td><td>R@10</td><td>R@1</td><td>R@5</td><td>R@10</td><td>R@1</td><td>R@5</td><td>R@10</td><td>R@1</td><td>R@5</td><td>R@10</td>
      </tr>
  	<tr align="center">
          <td width="120%">Wukong</td><td>51.7</td><td>78.9</td><td>86.3</td><td>77.4</td><td>94.5</td><td>97.0</td><td>76.1</td><td>94.8</td><td>97.5</td><td>92.7</td><td>99.1</td><td>99.6</td>
      </tr>
  	<tr align="center">
          <td width="120%">R2D2</td><td>60.9</td><td>86.8</td><td>92.7</td><td>84.4</td><td>96.7</td><td>98.4</td><td>77.6</td><td>96.7</td><td>98.9</td><td>95.6</td><td>99.8</td><td>100.0</td>
      </tr>
  	<tr align="center">
          <td width="120%">CN-CLIP</td><td>71.2</td><td>91.4</td><td>95.5</td><td>83.8</td><td>96.9</td><td>98.6</td><td>81.6</td><td>97.5</td><td>98.8</td><td>95.3</td><td>99.7</td><td>100.0</td>
      </tr>
  </table>
  <br>
  
  **COCO-CN Retrieval**:
  <table border="1" width="100%">
  	<tr align="center">
          <th>Task</th><th colspan="6">Text-to-Image</th><th colspan="6">Image-to-Text</th>
      </tr>
      <tr align="center">
          <th>Setup</th><th colspan="3">Zero-shot</th><th colspan="3">Finetune</th><th colspan="3">Zero-shot</th><th colspan="3">Finetune</th>
      </tr>
      <tr align="center">
          <td>Metric</td><td>R@1</td><td>R@5</td><td>R@10</td><td>R@1</td><td>R@5</td><td>R@10</td><td>R@1</td><td>R@5</td><td>R@10</td><td>R@1</td><td>R@5</td><td>R@10</td>
      </tr>
  	<tr align="center">
          <td width="120%">Wukong</td><td>53.4</td><td>80.2</td><td>90.1</td><td>74.0</td><td>94.4</td><td>98.1</td><td>55.2</td><td>81.0</td><td>90.6</td><td>73.3</td><td>94.0</td><td>98.0</