Solved by ptrblck in post #2 
              
              
                I guess you are missing the deviceGuard via: 
const at::cuda::OptionalCUDAGuard device_guard(device_of(tensor));

which would use the default device in your custom CUDA extension and will thus run into illegal memory accesses.