Hello, it is unclear to me what is the efficient way to run independent jobs (e.g., the many multiple runs of a hyper-parameter search effort) on a machine with multiple GPUs. I am not sure how Pytorch handles multiple GPUs, but I can see three ways with each possibly being better depending on how multiple GPUs are handled:

Run the jobs one by one serially on the machine
Run as many jobs in parallel on the machine, limiting each to a different GPU with CUDA_VISIBLE_DEVICES
Run as many jobs in parallel on the machine, without limiting runs to specific GPUs
Which of these options would be best? Or perhaps something else entirely?