ptrblck:

I don’t know which PyTorch, CUDA, cudnn versions and GPU


1.8.1, 11.2, 1080
The thing that bothers me is that without the register_non_leaf_grads everything works fine no complain about not being able to find cuDNN algo.
Also if I replace the register_nonleaf_grads for logits.retain_grad() things also work without any errors, which leads me to believe that there might be sth wrong with either register_nonleaf_grads or internally in pytorch? (I could be wrong though)