BERT layers are initialized as follows:

  
      github.com
  
  
    huggingface/transformers/blob/d5d2744aa799b94488960a261d1b7376d791a621/src/transformers/modeling_bert.py#L592-L602 89

def _init_weights(self, module):    """ Initialize the weights """    if isinstance(module, (nn.Linear, nn.Embedding)):        # Slightly different from the TF version which uses truncated_normal for initialization        # cf https://github.com/pytorch/pytorch/pull/5617        module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)    elif isinstance(module, nn.LayerNorm):        module.bias.data.zero_()        module.weight.data.fill_(1.0)    if isinstance(module, nn.Linear) and module.bias is not None:        module.bias.data.zero_()



  
  
    
    
  
  


You can set the seed before initializing the model, like so:
def set_seed(seed: Optional[int] = None):
    """Set all seeds to make results reproducible (deterministic mode).
       When seed is None, disables deterministic mode.
    :param seed: an integer to your choosing
    """
    if seed is not None:
        torch.manual_seed(seed)
        torch.cuda.manual_seed_all(seed)
        torch.backends.cudnn.deterministic = True
        torch.backends.cudnn.benchmark = False
        np.random.seed(seed)
        random.seed(seed)
        os.environ['PYTHONHASHSEED'] = str(seed)