Either use x = torch.tensor(x, requires_grad=True) (also works with other factory functions = things returning tensors) or use my_tensor.requires_grad = True (if it throws an error or doesn’t work, go back to one). The key here is to impose the requires_grad on a leave node that does has not been used in calculation yet.
Best regards
Thomas
P.S.: But on the original topic of the “most activated part of the feature matrix”: I think that gradient does not work terribly well for that due to the tanh nonlinearities have low gradient when they activate strongly (the gradient is small - known as saturation in the training context, bites here, too). There are people proposing a different relevance propagation that has “skip the nonlinearities during backward” as a key ingredient http://aclweb.org/anthology/W/W17/W17-5221.pdf 5 . In fact, I might have a demo of that in PyTorch to share in a while.