EDIT:
Thanks Victor for the additional thoughts. These two code snippets do work!
There is one issue though: I tried the functions you provided with a recent nightly build (1.10.0.dev20210805+cu102). Probably because of some incompatibility issues between this version and my cudatoolkit, the torch.autograd.grad() is much slower than before (in one instance, 40 seconds vs 5 seconds). Is there a PyTorch+cudatoolkit version combination that you recommend me to try? Thanks!