I implemented the very recent NeurIPS 21 paper Transformer in Transformer in TensorFlow which uses attention inside local patches essentially using pixel level attention paired with patch level attention. This also achieves SoTA performance on image classification beating ViT and DeiT with similar computational cost.

  
      

      GitHub
  

  
    

GitHub - Rishit-dagli/Transformer-in-Transformer: An Implementation of... 11

  An Implementation of Transformer in Transformer for image classification, attention inside local patches - GitHub - Rishit-dagli/Transformer-in-Transformer: An Implementation of Transformer in Tran...