Solved by jpeg729 in post #5 
              
              
                Weird. My implementation of the Inverse square root linear unit works fine. 
class ISRLU(torch.autograd.Function):
    @staticmethod
    def forward(ctx, tensor, alpha=1):
        negatives = torch.min(tensor, torch.Tensor([0]))
        nisr = torch.rsqrt(1. + alpha * (negatives ** 2))
        retur…