Solved by sebastienwood in post #2 
              
              
                I tried at initialisation : 
net = net.to(device).half()
for layer in net.modules():
    if isinstance(layer, nn.BatchNorm2d) or isinstance(layer, convmodule):
        layer.float()
        if isinstance(layer, convmodule): layer.fan_in.float()

I’m not sure if it’s legit but it’s a bit better on GP…