Thank you for your explanations!



 ptrblck:

Yes, ran it with the default setup first and compared it to benchmark mode, which didn’t yield a speedup.


How to setup pyorch with cudnn8? By compiling from source with system cudnn8 library setup?



 ptrblck:

It’s a cudnn “data gradient” kernel, which computes the gradient for the input activation.


Gradients are still computed even with torch.no_grad()?