backward takes as input the gradients to flow back. NOT what you want to differentiate.
You can do what you want in two ways (assuming feature_map has requires_grad=True):

call .retain_grad() on the feature map. Then when you call loss.backward(), your feature map will have a .grad attribute containing it’s gradients wrt the loss.
Use the autograd.grad() 52 for which you can specify which inputs you want gradients for and it will return them to you.