The original issue was created for PyTorch 0.3, so let’s focus on your new problem instead. 
Are you using any device-specific operations such as empty_cache without specifying the device id?
This could create a new CUDA context on the default device from another process, if you haven’t masked them via CUDA_VISIBLE_DEVICES.
If that’s not the case, could you post your code, so that we could have a look?