Okay, so after some digging:

This is expected behqviour
When you run load_model you re-load the original pretrained bert-base-uncased because you reinitialize the model. This line is run again: BertForSequenceClassification.from_pretrained(‘bert-base-uncased’

The message that you get is because “bert-base-uncased” contains the weights for BertForPretraining, which has the following submodules:


  
      github.com
  
  
    huggingface/transformers/blob/28cf873036d078b47fb9dd38ac3421a7c874da44/src/transformers/modeling_bert.py#L858-L865 2

class BertForPreTraining(BertPreTrainedModel):
    def __init__(self, config):
        super().__init__(config)

        self.bert = BertModel(config)
        self.cls = BertPreTrainingHeads(config)

        self.init_weights()



  
  
    
    
  
  


The warning that you get tells you that the original model has weights (heads) that BertForSequenceClassification does not have. The final classifier layer in BertForSequenceClassification is randomly initialized and does not get the weights from the pretrained modelùs seq_relationship. As said, this warning will return every time you use BertForSequenceClassification.from_pretrained("bert-base-uncased").
The correct way, and most pythonic, is to subclass the PreTrainedModel and use the save_pretrained and load_pretrained methods directly.