jienengchen
/

ViTamin-B-LTT

Feature Extraction

Transformers

PyTorch

custom_code

Model card Files Files and versions Community

bbexx commited on Apr 8, 2024

Commit

ca48de7

1 Parent(s): 9b46a92

update

Browse files

Files changed (1) hide show

README.md +5 -0

README.md CHANGED Viewed

@@ -22,6 +22,7 @@ from PIL import Image
 from transformers import AutoModel, CLIPImageProcessor
 device = "cuda" if torch.cuda.is_available() else "cpu"
 model = AutoModel.from_pretrained(
     'jienengchen/ViTamin-B-LTT',
     trust_remote_code=True).to(device).eval()
@@ -47,6 +48,10 @@ print("Label probs:", text_probs)
 | image encoder | image size | num patches | text encoder depth/width | seen samples (B)  | trainable params Image+Text (M) | MACs Image+Text (G)  | ImageNet Acc. | avg. 38 datasets | ImageNet dist. shift. | VTAB | retrieval |
 |---------------|------------|-------------|--------------------------|-------------------|---------------------------------|----------------------|---------------|------------------|-----------------------|------|-----------|
 | ViTamin-L     | 224        | 196         | 12/768                   | 12.8              | 333.3+123.7                     | 72.6+6.6             | 80.8          | 66.7             | 69.8                  | 65.3 | 60.3      |
 | ViTamin-L     | 256        | 256         | 12/768                   | 12.8+0.2          | 333.4+123.7                     | 94.8+6.6             | 81.2          | 67.0             | 71.1                  | 65.3 | 61.2      |
 | ViTamin-L     | 336        | 441         | 12/768                   | 12.8+0.2          | 333.6+123.7                     | 163.4+6.6            | 81.6          | 67.0             | 72.1                  | 64.4 | 61.6      |

 from transformers import AutoModel, CLIPImageProcessor
 device = "cuda" if torch.cuda.is_available() else "cpu"
+# obtained 70.72% zero-shot ImageNet score
 model = AutoModel.from_pretrained(
     'jienengchen/ViTamin-B-LTT',
     trust_remote_code=True).to(device).eval()
 | image encoder | image size | num patches | text encoder depth/width | seen samples (B)  | trainable params Image+Text (M) | MACs Image+Text (G)  | ImageNet Acc. | avg. 38 datasets | ImageNet dist. shift. | VTAB | retrieval |
 |---------------|------------|-------------|--------------------------|-------------------|---------------------------------|----------------------|---------------|------------------|-----------------------|------|-----------|
+| ViTamin-S     | 224        | 196         | 12/384                   | 1.28              | 22.0+40.4                       | 5.50+1.64            | 62.2          | 53.2             | 51.3	| 51.7	| 50.0 |
+| ViTamin-S-LTT  | 224       | 196         | 12/384                   | 1.28              | 22.0+40.4                       | 5.50+1.64            | 63.4          | 54.6             |  51.6 | 54.9 | 52.9 |
+| ViTamin-B     | 224        | 196         | 12/512                   | 1.28              | 87.5+63.4                       | 21.8+2.9             | 68.9          | 57.7             | 58.3	| 56.4	| 54.1 |
+| ViTamin-B-LTT  | 224       | 196         | 12/512                   | 1.28              | 87.5+63.4                       | 21.8+2.9             | 70.8          | 59.4             | 59.3 | 56.6 | 59.4 |
 | ViTamin-L     | 224        | 196         | 12/768                   | 12.8              | 333.3+123.7                     | 72.6+6.6             | 80.8          | 66.7             | 69.8                  | 65.3 | 60.3      |
 | ViTamin-L     | 256        | 256         | 12/768                   | 12.8+0.2          | 333.4+123.7                     | 94.8+6.6             | 81.2          | 67.0             | 71.1                  | 65.3 | 61.2      |
 | ViTamin-L     | 336        | 441         | 12/768                   | 12.8+0.2          | 333.6+123.7                     | 163.4+6.6            | 81.6          | 67.0             | 72.1                  | 64.4 | 61.6      |