Sure, you can just use any default model, e.g. BertModel and add your custom classification heads on top of that. Have a look at the existing iclassification implementation. You can basically duplicate that, but add another classifier layer. Of course you’ll also have to adapt the forward method accordingly.

  

      github.com
  

  
    huggingface/transformers/blob/c1e47bf4fe1d9de06d774cc2c24ec5a93461c5a5/src/transformers/models/bert/modeling_bert.py#L1481 1



      
      
                  )
          
          

          
@add_start_docstrings(
              """
              Bert Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled
              output) e.g. for GLUE tasks.
              """,
              BERT_START_DOCSTRING,
          )
          class BertForSequenceClassification(BertPreTrainedModel):
              def __init__(self, config):
                  super().__init__(config)
                  self.num_labels = config.num_labels
                  self.config = config
          
          
        self.bert = BertModel(config)
                  classifier_dropout = (
                      config.classifier_dropout if config.classifier_dropout is not None else config.hidden_dropout_prob
                  )
                  self.dropout = nn.Dropout(classifier_dropout)