This seems like a very specific use case and there is not a clear definition of how this would work. It would probably be easiest just to run loss.backward() and then zero out the particular gradients you don’t want to consider.