Hi @diravan, thanks for this nice solution! I was trying it just now but encounter one small problem, so I wanted to drop it here and see if you might be able to help out 
In my code I encountered the error that says RuntimeError: element 0 of variables tuple is volatile when I tried to execute the line in your code torch.autograd.grad. I didn’t know how to deal with it, but I checked that my output and input Variables do not require gradients, e.g., output.requires_grad = False and input.requires_grad = False. But even after I set those two Variables to require grad, I still get the above error.
Thanks in advance!