Apologies for the old thread resurrection here.
If I have a single machine with 2 mismatched GPUs (1 fast with big memory, 1 slow with small memory), DataParallel will only go as fast as the slow GPU with less memory, making it not worth using - as was covered above.
Does DistributedDataParallel on a single machine with 2 mismatched GPUs make sense? Or am I better off simply using the single fast GPU?
I’m trying to understand the usages of DistributedDataParallel (SPSD, SPMD, etc.)
I’ve been trying to read this and getting a bit confused: [POLL][RFC] Can we retire Single-Process Multi-Device Mode from DistributedDataParallel? · Issue #47012 · pytorch/pytorch · GitHub 1