[doc] parallelism - when to use which mode
# 🚀 Feature request

Was asked to expand https://huggingface.co/transformers/master/parallelism.html to include recommendations on which mode to use when.
@BramVanroy, please have a look if this addresses your question and I will add it to the doc. It of course assumes that https://huggingface.co/transformers/master/parallelism.html has been read (hence the abbreviations).

If more information is needed please don't hesitate to say what you feel is missing and how things can be improved. Thank you.

I just wasn't sure about single node / multi-gpu as I haven't played much with PP/TP on a single node.

------------
## Which Strategy To Use When

Here is a very rough outlook at which parallelism strategy to use when. The first on the list is typically faster.

**⇨ Single GPU**

* Model fits onto a single GPU:

    1. Normal use

* Model doesn't fit onto a single GPU:

    1. ZeRO + Offload CPU and optionally NVMe


**⇨ Single Node / Multi-GPU**

* Model fits onto a single GPU:

    1. DDP - Distributed DP
    2. ZeRO - may or may not be faster depending on the situation and configuration used

* Model doesn't fit onto a single GPU:

    1. ZeRO
    2. TP
    3. PP

    (not sure which one will be faster here - haven't done enough experiments)

**⇨ Multi-Node / Multi-GPU**

* When you have fast inter-node connectivity:

    1. ZeRO - as it requires close to no modifications to the model
    2. PP+TP+DP - less communications, but requires massive changes to the model

* when you have slow inter-node connectivity:

    1. DP+PP+TP+ZeRO
<|||||>This is already very useful for most people I think! I personally haven't even tried anything else but regular training and single node-muliti gpu DDP, but I can see how this small overview helps users. It makes it easier for them to "choose what to do".

Thanks!