Solved by albanD in post #6 
              
              
                The custom CUDA extension is differentiable 

How? 
The autograd only tracks torch functions. Your extension uses cuda directly yo compute the result. So it is not differentiable!