We’re currently adding a TF implementation of the Vision Transformer: Add TFViTModel by ydshieh · Pull Request #13778 · huggingface/transformers · GitHub 5
This will also make it easier to add the other vision models (DeiT, BEiT), which are very similar to ViT.