I guess you are missing the deviceGuard via:
const at::cuda::OptionalCUDAGuard device_guard(device_of(tensor));

which would use the default device in your custom CUDA extension and will thus run into illegal memory accesses.