That’s a good point; I hadn’t considered the disparity introduced by having a different number of epochs. While fixing up my testing script, I stumbled across what I think is a key culprit: thermal throttling.
My home setup has the two GPUs in adjacent slots, and what I think is happening is that the airflow into the top GPU is being warmed by the backplate of the bottom GPU, because if I heat the GPUs up with a job the top hits 90C and the pclck from nvidia-smi dmon drops, but it drops more with a 2 GPU job.
The hint that clued me in was the 2-GPU times getting worse as I increased the epoch size, rather than better.
While I had tested on work systems, those earlier tests might have suffered from issues with epoch counts, etc. I’m going to rerun those tests with my updated testing script on work systems and see what the results are. I’ll report back when I have them (probably tomorrow).
Thank you to everyone who took the time to read, comment, and/or run my testing script.