Hi @ptrblck,
I will work on getting the stacktrace. It would be much easier for me to load a preexisting CUDA installation through my SC institute than to use a Docker container, I don’t have much experience using Docker.
As for your second question, I am not using any CUDA extensions, I am solely using vanilla PyTorch code.
Edit: Here is the backtrace –
(cuda-gdb) target cudacore core_1627076158_cn2102_2891958.nvcudmp

Opening GPU coredump: core_1627076158_cn2102_2891958.nvcudmp

[New Thread 2892012]

CUDA Exception: Warp Illegal Address

The exception was triggered at PC 0x561e719946f0

[Current focus set to CUDA kernel 0, grid 1263657, block (0,0,0), thread (0,0,0), device 0, sm 0, warp 3, lane 0]

#0 0x0000561e71994720 in void laswp_kernel2<double, false>(int, double*, unsigned long, int, int, int const*, int)<<<(1,1,1),(4,64,1)>>> ()

(cuda-gdb) bt

#0 0x0000561e71994720 in void laswp_kernel2<double, false>(int, double*, unsigned long, int, int, int const*, int)<<<(1,1,1),(4,64,1)>>> ()

I think it spit out the backtrace automatically, but I used the bt command once more just to be sure. This was generated with a loaded local installation of cuda-gdb that came with CUDA 11.2
Probably should have given this information upfront, but all my training was conducted on a single V100 GPU