Which Strategy To Use When
Here is a very rough outline at which parallelism strategy to use when. The first on each list is typically faster.
⇨ Single GPU
Model fits onto a single GPU:
Normal use
Model doesn't fit onto a single GPU:
ZeRO + Offload CPU and optionally NVMe
as above plus Memory Centric Tiling (see below for details) if the largest layer can't fit into a single GPU
Largest Layer not fitting into a single GPU:
ZeRO - Enable Memory Centric Tiling (MCT). It allows you to run arbitrarily large layers by automatically splitting them and executing them sequentially. MCT reduces the number of parameters that are live on a GPU, but it does not affect the activation memory. As this need is very rare as of this writing a manual override of torch.nn.Linear needs to be done by the user.
⇨ Single Node / Multi-GPU
