How does a commoner train a Transformers-based model on small and medium datasets like CIFAR-10, ImageNet-1k and still attain competition results? What if they don’t have the luxury of using a modern GPU cluster or TPUs?
You use Compact Convolutional Transformers (CCT). In this example, I walk you through the concept of CCTs and also present their implementation in Keras demonstrating their performance on CIFAR-10:

  
      

      keras.io
  

  
    

Keras documentation: Compact Convolutional Transformers 45



  

  
    
    
  

  


A traditional ViT model would take about 4 Million parameters to get to 78% on CIFAR-10 with 100 epochs. CCTs would take 30 epoch and 0.4 Million parameters to get there