Have you considered a smaller model, such as DistilBERT?
Huggingface have a DistilBERT version for Tensorflow, TFDistilBertForSequenceClassification, https://huggingface.co/transformers/model_doc/distilbert.html 2
When you use a pretrained BERT-type model, Yes, you are doing transfer learning.