Have you looked at using the keras-bert library (instead of using huggingface transformers library)?

  
      
      GitHub
  
  
    

CyberZHG/keras-bert 12

Implementation of BERT that could load official pre-trained models for feature extraction and prediction - CyberZHG/keras-bert


  
  
    
    
  
  


keras-bert works with TF2.   I used it to load a bert model from the tensorflow hub into colab.   I copied it onto my google drive first, and loaded from that copy, but I believe it is also possible to load directly using
!wget -q https://storage.googleapis.com/bert_models/2018_10_18/uncased_L-12_H-768_A-12.zip 2
!unzip -o uncased_L-12_H-768_A-12.zip
So far as I can tell, models loaded with keras-bert are not compatible with transformers.  I tried to get my fine-tuned model into transformers format (because I wanted to use Jesse Vig’s visualisations), but I never got it to work.
I believe there is something about the way the transformers code is written that means you can’t use non-transformers models in place of transformers models, or at least not without some serious poking of the python.   However, I could be mistaken.