Okay - this is a continuation of the message stream I had with @Nikronic, and is my solution to calculating the gain in order to properly use PReLU nonlinearity. I have not yet implemented this myself, but it’s what makes sense to me after reading the derivation in the original paper:  https://arxiv.org/pdf/1502.01852.pdf 1. I’m writing this both for the community and my own later use 
So, we start from the std definition in the paper:

If we rearrange this, we obtain that the standard deviation (=sqrt(Var)) is given by the following two, which are the values obtained from https://pytorch.org/docs/stable/nn.init.html. The gain is basically identically calculated to the one in LeakyReLU.

Screenshot 2020-06-26 at 15.51.54389×515 12 KB
The equation derived in the original paper is the one used for the normal distribution, not the uniform one  torch.nn.init.kaiming_normal_ . So, in theory, we could just use this one. However, this is not possible, as the kaiming_normal_ function in PyTorch calls torch.nn.init.calculate_gain which does not accept PReLU as a nonlinearity. Thus, we need a workaround this issue:
The alternative is to just calculate our own standard deviation, which is actually easier than I thought. In the paper, they suggest initiating the negative_slope to whatever value we also use to initiate it in our PReLU. For PyTorch, that would be 0.25 (https://pytorch.org/docs/stable/nn.html#prelu 1). We also need to calculate the fan_mode, for which we can look at how this is calculated in the PyTorch source (https://pytorch.org/docs/stable/modules/torch/nn/init.html#kaiming_normal 3):
def _calculate_fan_in_and_fan_out(tensor):
    dimensions = tensor.dim()
    if dimensions < 2:
        raise ValueError("Fan in and fan out can not be computed for tensor with fewer than 2 dimensions")

    num_input_fmaps = tensor.size(1)
    num_output_fmaps = tensor.size(0)
    receptive_field_size = 1
    if tensor.dim() > 2:
        receptive_field_size = tensor[0][0].numel()
    fan_in = num_input_fmaps * receptive_field_size
    fan_out = num_output_fmaps * receptive_field_size

    return fan_in, fan_out

Following the calcualtion of the std, and knowing we have a mean of 0, we can rewrite the code provided here (Example 10 - https://www.programcreek.com/python/example/107693/torch.nn.PReLU) to produce our required weight initialisation:
        for m in self.modules():
            if isinstance(m, nn.Conv2d):
                n = fan_in
                negative_slope = 0.25
                m.weight.data.normal_(0, math.sqrt(2. / (n * (1 + negative_slope ** 2)) ) )
            elif isinstance(m, nn.BatchNorm2d):
                m.weight.data.fill_(1)
                m.bias.data.zero_() 

Of course, if one uses to use fan_out, they can just replace it using the code above.
@Nikronic - does what I wrote above make sense? Asking as I think you have more experience with these things than I do.