Update (I hope it might help somebody):
i started to write my own C++/CUDA extension, along with a small C++ starter. I want to use that starter for easier debugging and profiling using cuda_gdb and friends. After making everything work with the cmake build system, I started up nsight to step into the code. Right after calling Tensor::cuda(), the debugger hanged. I thought that something is not working, but turns out that it just takes one minute and 40 seconds for that one step. After that stepping is back to normal. I guess that loading all these kernels inside cuda_gdb is super slow.
Similar things were also reported in the nvidia forums (thought that was 2009): https://forums.developer.nvidia.com/t/cuda-gdb-performance/9175 14