Hi @theudster, I’m pretty sure that ProtBERT has a CLS token since you can see it in the tokenizer’s special tokens map:
from transformers import AutoTokenizer, AutoModelForMaskedLM

tokenizer = AutoTokenizer.from_pretrained("Rostlab/prot_bert")
# returns {'unk_token': '[UNK]', 'sep_token': '[SEP]', 'pad_token': '[PAD]', 'cls_token': '[CLS]', 'mask_token': '[MASK]'}
tokenizer.special_tokens_map

You can also see it by encoding a text and then decoding it:
text = "I love Adelaide!"
# add_special_tokens=True is set by default
text_enc = tokenizer.encode(text)

for tok in text_enc:
    print(tok, tokenizer.decode(tok))

You say you froze the BERT layers, so I’m wondering how you’re doing fine-tuning? I’ve sometimes found that the tutorials in the docs aren’t always complete, so for fine-tuning with text classification I would recommend following Sylvain’s tutorial here: https://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/token_classification.ipynb 9