Hi am using this 35 implementation of DeepLab for PyTorch and would like to perform hyperparameter optimization by training multiple models at the same time.
This implementation uses nn.DataParallel to train a model on multiple GPUs.
I start one training process after executing export CUDA_VISIBLE_DEVICES=0,1. When I want to start the other training process, I get two different errors depending on the GPU ids in CUDA_VISIBLE_DEVICES.
If I type export CUDA_VISIBLE_DEVICES=0,1,2,3 and train the second model on GPU 2 and 3, I receive:
RuntimeError: module must have its parameters and buffers on device cuda:2 (device_ids[0]) but found one of them on device: cuda:0
On the other hand if I execute export CUDA_VISIBLE_DEVICES=2,3 I receive AssertionError: Invalid device id.
How can I train two models at the same time while another process already loaded the input data in a GPU.
Do I have to specify on which device the model should run using .to_device(cuda:x)?