Option 1 seems bad because you’re not exploiting the potential parallelism.
Option 3 is interesting. I’m not sure how pytorch picks which GPU to use if you don’t specify but I wouldn’t be surprised if everything ran on GPU 0 or something.
Option 2 (using CUDA_VISIBLE_DEVICES) looks good to me. Pytorch has this setDevice API where you can tell it what gpu to put each tensor on, but this code 50 seems to recommend just using CUDA_VISIBLE_DEVICES