Your custom loss function using numpy should detach the loss from the computation graph, so that all PyTorch parameters, which were used before detaching won’t get a gradient.
Could you check it by printing the grad attribute of some parameters after calling backward?