I’m unsure if you can set the default device globally, but you could use the CUDAGuard() inside your method instead:
CUDAGuard device_guard(device);

which makes sure to use the specified device id.
(Thanks @tom for clarification  )