I assume cre_loss gives a proper loss tensors with a valid grad_fn, while loss_arr contains the loss using numpy operations. If that’s the case, you would add constants to loss, which won’t change the gradients.



 Mike_Deng:

Is there a way to correctly implement this? I can’t really think of a way to implement this with differentiable operations.


The first step would be to replace all numpy operations with PyTorch ones. If some operations are not differentiable, you could think about an approximation function and implement this instead.