Note that your call to autograd.grad() in the first post is not correct. You should give the input before the grad_output.
You can do:
X = torch.tensor(self.inputs, dtype=torch.float, device=device, requires_grad=True)

x = phi(X)

d = x - X

F, = autograd.grad(d, X, ones)

Is that what your code looks like?