Tanh --> Xavier
ReLU --> He

No that’s not correct, PyTorch’s initialization is based on the layer type, not the activation function (the layer doesn’t know about the activation upon weight initialization).
For the linear layer, this would be somewhat similar to He initialization, but not quite:

  
      github.com
  
  
    pytorch/pytorch/blob/9e2f2cab94027c1be1860b9b5e98ac13c6b0516e/torch/nn/modules/linear.py#L48-L52 119

def reset_parameters(self):
    stdv = 1. / math.sqrt(self.weight.size(1))
    self.weight.data.uniform_(-stdv, stdv)
    if self.bias is not None:
        self.bias.data.uniform_(-stdv, stdv)



  
  
    
    
  
  


def reset_parameters(self):
    stdv = 1. / math.sqrt(self.weight.size(1))
    self.weight.data.uniform_(-stdv, stdv)
    if self.bias is not None:
        self.bias.data.uniform_(-stdv, stdv)

Ie., when I remember correctly, He init is “sqrt(6 / fan_in)” whereas in PyTorch Linear layer it’s “1. / sqrt(fan_in)”