Thank you. As soon as I didn’t use CUDA_VISIBLE_DEVICES and specified ‘.to(device)’ I can train multiple models, each on multiple GPUs, at the same time.
It looks like _check_balance(device_ids) somehow gets the device id 0 even though nn.DataParallel(model, device_ids=[2,3]) has GPU IDs 2 and 3. That was the problem why using CUDA_VISIBLE_DEVICES=2,3 was resulted in an AssertionError: Invalid device id .