Model fits onto a single GPU:
DDP - Distributed DP
ZeRO - may or may not be faster depending on the situation and configuration used
Model doesn't fit onto a single GPU:
PP
ZeRO
TP
With very fast intra-node connectivity of NVLINK or NVSwitch all three should be mostly on par, without these PP will be faster than TP or ZeRO. The degree of TP may also make a difference. Best to experiment to find the winner on your particular setup.
TP is almost always used within a single node. That is TP size <= gpus per node.
Largest Layer not fitting into a single GPU:
If not using ZeRO - must use TP, as PP alone won't be able to fit.
With ZeRO see the same entry for "Single GPU" above
⇨ Multi-Node / Multi-GPU
When you have fast inter-node connectivity:
ZeRO - as it requires close to no modifications to the model
