Hey. I realise now that my answer wasn’t clear earlier. They’ve used the Kaiming uniform initialisation 1, yes but they’ve chosen the parameter a such that gain is set to be 1/sqrt(3). The way they’ve done that is using the calculate_gain function (pytorch/init.py at 4ff1823fac8833d4bdfe333c49fd486143175395 · pytorch/pytorch · GitHub 3). With nonlinearity=leaky_relu, the gain factor is math.sqrt(2.0 / (1 + param ** 2)). Choose a=param=math.sqrt(5). Then the gain factor is 1/sqrt(3).