Hi guys, I’m running a WRN 17 implementation with DataParallel with 4 GPUs. I’m seeing only about 50% reduction in epoch time. Is this normal? GPU utilizations are between 88 to 98 percent, which doesn’t seem too bad.
What kind of speedup do you expect from a 4GPU setup with DataParallel, on a reasonably large model like WRN?
Does it have anything to do with SLI setups? I’m running on Google Cloud Platform and I’ve no idea how they setup their GPUs.