Could you link to a reference implementation, which shows the desired behavior?
If I’m not mistaken, the perceptual loss is calculated by using a norm between the intermediate activations between two input images, and summing it to a final loss.
If my understanding is correct, I don’t see where the gradients can be used at this point, as they aren’t calculated yet.