You can manage internally (via torch commands) which gpu to use at any time.
Most of the data parallel funcs allows to set that and you can set the devices manually anyway 
Just mentioning that defining cuda_visible_devices inside python won’t work no matter what u do.