Yes, the tokenizers in transformers add the special tokens by default (see the docs here 22).
I’m not familiar with ProtBERT but I’m surprised its crashing Colab because the repo has some Colab examples: ProtTrans/ProtBert-BFD-FineTuning-MS.ipynb at master · agemagician/ProtTrans · GitHub 1
If you’re still having problems fine-tuning with a GPU, perhaps you can reduce the batch size to avoid the OOM errors