Thanks @Nikronic! This makes things slightly more clear to me. However, I still have some questions, if that’s okay?
I understand the use of math.sqrt(5) and how this ties in with the nonlinearity, IF the nonlinearity is ReLU or LeakyReLU. In my case, I am using a PReLU non-linearity for now. This is similar to LeakyReLU, and the Kaiming Loss has been created for it. Given this, would the math.sqrt(5) still be a good choice for the a-parameter in Kaiming Loss?
Also, thank you for the advise on reinitilisation!