Solved by ngimel in post #16 
              
              
                The solution is simple, use torch.backends.cudnn.benchmark=True. cudnn heuristics picks truly atrocious algorithm for couple layers (one of the kernels takes 600 ms, another 50 ms). When cudnn is actually forced to benchmark and pick the best algo, the hessian computation speed becomes reasonable.