The very moment you convert a torch tensor to a numpy array, the autograd of pytorch loses track on the contents of the tensor. Unless you can write an analytical or numerical gradient calculation for the computation happening in numpy, this kind of interference will break the computation graph and its not possible to optimize the network.