I am hyperparameter searching and want to utilize all GPU’s with DistributedDataParallel. Let’s say all my distributed training code is in main.py. I want to test different hyperparmeters in a bash script like this so that I can train both models at once:
python main.py --param1 45 --param2 20 &
python main.py --param1 33 --param2 41

Should I expect any funny business from DistributedDataParallel training like this?