I think the question is a bit too vague but if you mean the BERT-like family of models can be loaded using the code above and similarly for others but ConvBERT for example has its own class (see here: ConvBERT — transformers 4.7.0 documentation 2).
The head that will be initialized will be of random weights so the model will need fine-tuning.