The argument device_ids from DataParallel are there to take care of the devices for you. Otherwise, you can try to define the environment variable CUDA_VISIBLE_DEVICES=2,3 to make sure that your process will never see the other devices.