sgugger sayakpaul HF staff commited on
Commit
c5f1ba7
1 Parent(s): 533645b

Create README.md (#1)

Browse files

- Create README.md (086e2580f08dd6806d05857315c8d271223bc45f)


Co-authored-by: Sayak Paul <[email protected]>

Files changed (1) hide show
  1. README.md +75 -0
README.md ADDED
@@ -0,0 +1,75 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ tags:
4
+ - vision
5
+ datasets:
6
+ - imagenet-1k
7
+ ---
8
+
9
+ # Vision Transformer (base sized model) pre-trained with MSN (patch size of 4)
10
+
11
+ Vision Transformer (ViT) model pre-trained using the MSN method. It was introduced in the paper [Masked Siamese Networks for Label-Efficient Learning](https://arxiv.org/abs/2204.07141) by Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Michael Rabbat, Nicolas Ballas and first released in [this repository](https://github.com/facebookresearch/msn).
12
+
13
+ Disclaimer: The team releasing MSN did not write a model card for this model so this model card has been written by the Hugging Face team.
14
+
15
+ ## Model description
16
+
17
+ The Vision Transformer (ViT) is a transformer encoder model (BERT-like). Images are presented to the model as a sequence of fixed-size patches.
18
+
19
+ MSN presents a joint-embedding architecture to match the prototypes of masked patches with that of the unmasked patches. With this setup, their method yields excellent performance in the low-shot and extreme low-shot regimes.
20
+
21
+ By pre-training the model, it learns an inner representation of images that can then be used to extract features useful for downstream tasks: if you have a dataset of labeled images for instance, you can train a standard classifier by placing a linear layer on top of the pre-trained encoder.
22
+
23
+ ## Intended uses & limitations
24
+
25
+ You can use the raw model for downstream tasks like image classification. See the [model hub](https://huggingface.co/models?filter=vit_msn) to look for different versions of MSN pre-trained models that interest you. The model is particularly beneficial when you have a few labeled samples in your training set.
26
+
27
+ ### How to use
28
+
29
+ Here is how to use this backbone encoder:
30
+
31
+ ```python
32
+ from transformers import AutoFeatureExtractor, ViTMSNModel
33
+ import torch
34
+ from PIL import Image
35
+ import requests
36
+
37
+ url = "http://images.cocodataset.org/val2017/000000039769.jpg"
38
+ image = Image.open(requests.get(url, stream=True).raw)
39
+
40
+ feature_extractor = AutoFeatureExtractor.from_pretrained("facebook/vit-msn-base-4")
41
+ model = ViTMSNModel.from_pretrained("facebook/vit-msn-base-4")
42
+ inputs = feature_extractor(images=image, return_tensors="pt")
43
+ with torch.no_grad():
44
+ outputs = model(**inputs)
45
+ last_hidden_states = outputs.last_hidden_state
46
+ ```
47
+
48
+ For fine-tuning on image classification use the `ViTMSNForImageClassification` class:
49
+
50
+ ```python
51
+ from transformers import AutoFeatureExtractor, ViTMSNForImageClassification
52
+ import torch
53
+ from PIL import Image
54
+ import requests
55
+
56
+ url = "http://images.cocodataset.org/val2017/000000039769.jpg"
57
+ image = Image.open(requests.get(url, stream=True).raw)
58
+
59
+ feature_extractor = AutoFeatureExtractor.from_pretrained("facebook/vit-msn-base-4")
60
+ model = ViTMSNForImageClassification.from_pretrained("facebook/vit-msn-base-4")
61
+
62
+ ...
63
+ ```
64
+
65
+
66
+ ### Citation
67
+
68
+ ```bibtex
69
+ @article{assran2022masked,
70
+ title={Masked Siamese Networks for Label-Efficient Learning},
71
+ author={Assran, Mahmoud, and Caron, Mathilde, and Misra, Ishan, and Bojanowski, Piotr, and Bordes, Florian and Vincent, Pascal, and Joulin, Armand, and Rabbat, Michael, and Ballas, Nicolas},
72
+ journal={arXiv preprint arXiv:2204.07141},
73
+ year={2022}
74
+ }
75
+ ```