It seems like I should be able to make all of the losses into a tensor and then take the gradient with respect to all parameters (essentially a Jacobian), and then just choose the relevant gradients. But it doesn’t seem that PyTorch has a good way of computing Jacobians from what I’ve read so far.