I would like to run 2 models (each model runs on one gpu) in one script, I can run them in subprocesses and send model and tensors to a specific gpu by  tensor.cuda(device_id) , so that these two models runs independently. 2 models are created successfully, and I can see they take gpu memory respectively. During inference, I got the following error:
transform: failed to synchronize: cudaErrorIllegalAddress: an illegal memory access was encountered

I guess the problem may come from: for some functions it runs in the default device, and I have to write  torch.cuda.set_device()  to set the global(default) device, how can I work around this to run 2 model instances on each gpu?