DeepSpeed: There is no obvious benefit from increasing the number of GPU nodes
I tested the performance of DeepSpeed on a GPU cluster, all GPUs are V100 16GB.
"2x4" means 2 node with 4 gpu each

my setup:

HF launch scripts:
```bash
deepspeed \
    --hostfile=hostfile.txt \
    run_clm.py \
    --deepspeed ds_config.json \
    --model_name_or_path /huggingface_with_deepspeed/gpt2-large/ \
    --preprocessing_num_workers $(nproc) \
    --do_train \
    --train_file /data/train_mini.txt \
    --fp16 true \
    --output_dir output \
    --overwrite_output_dir true \
    --per_device_train_batch_size 1
```

DeepSpeed config:
```json
{
    "train_micro_batch_size_per_gpu": "auto",
    "fp16": {
        "enabled": true,
        "loss_scale": 0
    },

    "flops_profiler": {
        "enabled": true,
        "profile_step": 10,
        "module_depth": -1,
        "top_modules": 1,
        "detailed": true,
        "output_file": "./tmp.log"
      },

    "zero_optimization": {
        "stage": 3,
        "offload_optimizer": {
            "device": "cpu",
            "pin_memory": true
        },
        "offload_param":{
            "device": "cpu",
            "pin_memory": true
        },
        "allgather_partitions": true,
        "allgather_bucket_size": 2e8,
        "reduce_scatter": true,
        "reduce_bucket_size": 2e8,
        "overlap_comm": true,
        "contiguous_gradients": true,
        "grad_hooks": true,
        "round_robin_gradients": false
    },

    "gradient_clipping": 1.0,

    "wall_clock_breakdown": false,

    "sparse_attention": {
    "mode": "fixed",
    "block": 16,
    "different_layout_per_head": true,
    "num_local_blocks": 4,
    "num_global_blocks": 1,
    "attention": "bidirectional",
    "horizontal_global_attention": false,
    "num_different_global_patterns": 4
  }
}
```

performance result:


|GPT2-large(770M)<br>BATCH_SIZE=1<br>1x4 | HF with DS(ZeRO-1) | HF with DS(ZeRO-2) | HF with DS(ZeRO-2-offload) | HF with DS(ZeRO-3) | HF with DS(ZeRO-3-offload)|
|-- | -- | -- | -- | -- | --|
|CUDA MEM(MB)|14837 | 15983 | 14483 | 16155 | 13145|
|Forward time(ms)|83.59 | 83.72 | 83.47 | 529.49 | 538.30|
|Backward time(ms)|437.23 | 283.19 | 658.34 | 1016.79 | 976.68|
|Step time(ms)|2370.66 | 2372.91 | 4455.99 | 154.37 | 2163.18|
|Total time(ms)|2892 | 2740 | 5199 | 1770 | 3740|
|Throughput(samples/s)|1.38 | 1.46 | 0.77 | 2.26 | 1.06|

|GPT2-large(770M)<br>BATCH_SIZE=1<br>2x4 | HF with DS(ZeRO-1) | HF with DS(ZeRO-2) | HF with DS(ZeRO-2-offload) | HF with DS(ZeRO-3) | HF with DS(ZeRO-3-offload)|
|-- | -- | -- | -- | -- | --|
|CUDA MEM(MB)|14979 | 14903 | 14457 | 14521 | 13147|
|Forward time(ms)|83.89 | 83.90 | 83.61 | 2685.72 | 2701.03|
|Backward time(ms)|5370.95 | 2783.34 | 3180.92 | 4502.87 | 4618.15|
|Step time(ms)|2370.66 | 2445.77 | 2338.42 | 3759.82 | 131.51 | 1335.59|
|Total time(ms)|2892 | 7901 | 5206 | 7063 | 7320 | 8655|
|Throughput(samples/s)|1.01 | 1.53 | 1.13 | 1.09 | 0.92|

|GPT2-large(770M)<br>BATCH_SIZE=1<br>4x4 | HF with DS(ZeRO-1) | HF with DS(ZeRO-2) | HF with DS(ZeRO-2-offload) | HF with DS(ZeRO-3) | HF with DS(ZeRO-3-offload)|
|-- | -- | -- | -- | -- | --|
|CUDA MEM(MB)|14793 | 14401 | 14247 | 13763 | 12981|
|Forward time(ms)|83.36 | 83.43 | 83.71 | 2339.03 | 2414.61|
|Backward time(ms)|4980.09 | 2842.94 | 3194.67 | 4201.04 | 4348.77|
|Step time(ms)|2632.80 | 2777.44 | 3384.50 | 177.98 | 870.95|
|Total time(ms)|2892 | 7701 | 5707 | 6665 | 6722 | 7634|
|Throughput(samples/s)|2.07 | 2.80 | 2.4 | 2.38 | 2.09|


To my knowledge, there are something strange:
- GPU memory has not decreased significantly with the increase in the number of nodes ？
- Offload seem not reduce memory significantly？
- Refer to paper: [ZeRO-Offload](https://arxiv.org/abs/2101.06840), "It can train models with over 13 billion parameters on a single GPU with ZeRO2-Offload",  but the reality is that I can’t even train the GPT2-xl model on 16 GPUs, which is only 1.5B parameters

Are these results in line with expectations？
Thank you for your detailed report, @dancingpipi 

I will try to study your report later today, but until then please have a look at:
https://deepspeed.readthedocs.io/en/stable/memory.html
which should answer at least part of this query.<|||||>Your reported gpu memory usage numbers indeed don't make sense, the differences typically are quite dramatic. You can refer to the link above to see how it's supposed to be. How are you measuring the cuda memory? What do these tables report - an average? a total? on which gpu are you measuring this, which tool?

Are you sure your hostfile changes between the runs?

Did you know the Trainer has the memory measurements already built in?

Just add `--skip_memory_metrics 0` to the script's cmd - note: it reports just the first gpu's memory.<|||||>thanks for your quick reply!

> How are you measuring the cuda memory?

I measure the cuda memory with "nvidia-smi".

> What do these tables report - an average? a total?

An average

> on which gpu are you measuring this?

On first GPU

> which tool?

DeepSpeed print some metrics on the screen, look like this:
![image](https://user-images.githubusercontent.com/20511825/133878537-c98070d1-f15a-4b82-97c6-b85c172831db.png)

> Did you know the Trainer has the memory measurements already built in?

No,  I'll try the '--skip_memory_metrics 0'  argument<|||||>Let's ignore other metrics as they are irrelevant to this inquiry - let's stick to just memory measurement.

`nvidia-smi` may not always report the real memory usage for a given application, though it's great to see the overall memory availability. `nvml` and tools based on it are easier to use since they are programmable. 

In HF Transformers we use pytorch's internal memory management tools which are very precise and account only pytorch's allocations:
https://github.com/huggingface/transformers/blob/b518aaf193938247f698a7c4522afe42b025225a/src/transformers/trainer_utils.py#L276

----------------

One other concern I have is that Deepspeed doesn't validate its config file, so if you make a typo it just ignores the bad entry and uses its defaults. So looking at the log files helps to see that your config actually made through.

Also you have shared just one of the config files, but your table calls for many different configs so I can't see what you're doing.

Let's narrow the experiment a bit and just compare 2 configs on 1x 4 gpus, let's pick zero2+cpu_offload and zero3+cpu_offload.

So let's see the 2 configs and the 2 log files. For the log files please attach them to your comment via attachment, don't paste them, as they are huge, the 2 configs you can paste.

Thanks.

So it'd probably help to look at a few log files. For example, one log file with zero3 and another with zero2<|||||>**zero2_config.json**
```
{
    "train_micro_batch_size_per_gpu": "auto",
    "fp16": {
        "enabled": true,
        "loss_scale": 0
    },

    "flops_profiler": {
        "enabled": true,
        "profile_step": 10,
        "module_depth": -1,
        "top_modules": 1,
        "detailed": true,
        "output_file": "./tmp.log"
      },

    "zero_optimization": {
        "stage": 2,
        "offload_optimizer": {
            "device": "cpu",
            "pin_memory": true
        },
        "allgather_partitions": true,
        "allgather_bucket_size": 2e8,
        "reduce_scatter": true,
        "reduce_bucket_size": 2e8,
        "overlap_comm": true,
        "contiguous_gradients": true,
        "grad_hooks": true,
        "round_robin_gradients": false
    },

    "gradient_clipping": 1.0,

    "wall_clock_breakdown": false,

    "sparse_attention": {
    "mode": "fixed",
    "block": 16,
    "different_layout_per_head": true,
    "num_local_blocks": 4,
    "num_global_blocks": 1,
    "attention": "bidirectional",
    "horizontal_global_attention": false,
    "num_different_global_patterns": 4
  }
}
```

**zero3_config.json**

  ```
  "train_micro_batch_size_per_gpu": "auto",
    "fp16": {
        "enabled": true,
        "loss_scale": 0
    },

    "flops_profiler": {
        "enabled": true,
        "profile_step": 10,
        "module_depth": -1,
        "top_modules": 1,
        "detailed": true,
        "output_file": "./tmp.log"
      },

    "zero_optimization": {
        "stage": 3,
        "offload_optimizer": {
            "device": "cpu",
            "pin_memory": true
        },
        "offload_param":{
            "device": "cpu",
            "pin_memory": true
        },
        "allgather_partitions": true,
        "allgather_bucket_size": 2e8,
        "reduce_scatter": true,
        "reduce_bucket_size": 2e8,
        "overlap_comm": true,
        "contiguous_gradients": true,
        "grad_hooks": true,
        "round_robin_gradients": false
    },

    "gradient_clipping": 1.0,

    "wall_clock_breakdown": false,

    "sparse_attention": {
    "mode": "fixed",
    "block": 16,
    "different_layout_per_head": true,
    "num_local_blocks": 4,
    "num_global_blocks": 1,
    "attention": "bidirectional",
    "horizontal_global_attention": false,
    "num_different_global_patterns": 4
  }
}
```

[zero2.log](https://github.com/huggingface/transformers/files/7224608/zero2.log)
[zero3.log](https://github.com/huggingface/transformers/files/7224609/zero3.log)

@stas00 I'm sorry for not having timely feedback.
Above are config and log for zero2 and zero3.

I didn’t run the entire training because it would take a lot of time.<|||||>Thank you for sharing the configs and matching logs, @dancingpipi. The logs look matching the config, can't see any issue there.

> I didn’t run the entire training because it would take a lot of time.

but we need the metric reports - could you please run like 5 iterations of each of these 2 configs and include the full log with the final metric report?

And please include `--skip_memory_metrics 0` argument so we get the memory usage reported.

Additionally, I meant to ask why were you adding the sparse attention config? `transformers` isn't doing anything with it at the moment - perhaps it configures something there that adds up memory usage?<|||||>@stas00 Sorry for late reply.
I have run an entire training on a small dataset.

The logs are here.
[zero-2.log](https://github.com/huggingface/transformers/files/7282786/zero-2.log)
[zero-3.log](https://github.com/huggingface/transformers/files/7282868/zero-3.log)

<|||||>Thank you for sharing the logs, @dancingpipi, so let's have a look:

```
  # zero 2
  train_mem_cpu_alloc_delta  =     4429MB
  train_mem_cpu_peaked_delta =     2950MB
  train_mem_gpu_alloc_delta  =     1635MB
  train_mem_gpu_peaked_delta =     8862MB

  # zero 3
  train_mem_cpu_alloc_delta  =     3428MB
  train_mem_cpu_peaked_delta =     1442MB
  train_mem_gpu_alloc_delta  =     -399MB
  train_mem_gpu_peaked_delta =    10352MB
```

You can clearly see that memory usage is far from being the same.

The report with negative numbers is confusing under ZeRO3, so I made an improved reporting here: https://github.com/huggingface/transformers/pull/13915 so that it reports the initial memory at the start and now the negative delta adds up to the pre-allocated memory on the gpu. You can retest with that PR to get a more exact picture.

So let's step aside for a moment from your configs and use the default configs I pre-made for z2 and z3 and a small model:

ZeRO2
```
#MODEL="sshleifer/tiny-gpt2"
MODEL="gpt2"

PYTHONPATH=src deepspeed --num_gpus 2 \
    examples/pytorch/language-modeling/run_clm.py \
    --deepspeed tests/deepspeed/ds_config_zero2.json \
    --model_name_or_path $MODEL \
    --do_train \
    --train_file tests/fixtures/sample_text.txt \
    --fp16 true \
    --output_dir output \
    --overwrite_output_dir 1 \
    --per_device_train_batch_size 1 \
    --report_to none \
    --skip_memory_metrics 0

***** train metrics *****
  before_init_mem_cpu        =     5759MB
  train_mem_cpu_alloc_delta  =     1952MB
  train_mem_cpu_peaked_delta =      224MB


  before_init_mem_gpu        =        0MB
  train_mem_gpu_alloc_delta  =      324MB
  train_mem_gpu_peaked_delta =     2611MB


```
ZeRO3
```
MODEL="gpt2"

PYTHONPATH=src deepspeed --num_gpus 2 \
    examples/pytorch/language-modeling/run_clm.py \
    --deepspeed tests/deepspeed/ds_config_zero3.json \
    --model_name_or_path $MODEL \
    --do_train \
    --train_file tests/fixtures/sample_text.txt \
    --fp16 true \
    --output_dir output \
    --overwrite_output_dir 1 \
    --per_device_train_batch_size 1 \
    --report_to none \
    --skip_memory_metrics 0


***** train metrics *****
  before_init_mem_cpu        =     5861MB
  train_mem_cpu_alloc_delta  =     2158MB
  train_mem_cpu_peaked_delta =      224MB


  before_init_mem_gpu        =      193MB
  train_mem_gpu_alloc_delta  =     -108MB
  train_mem_gpu_peaked_delta =     2015MB
```

I rearranged the reports in groups so it's easy to make sense of the numbers.

It's easy to see that the memory usage is far from the same between the 2 cases.

Does this give any help? Perhaps you can build a bridge between what I shared and what you use and continue monitoring memory usage?

HF Trainer metrics use pytorch reporting - so you get the exact allocations for GPU memory usage.<|||||>I also run z3 w/ gpt2 on 1 vs 2 gpus and indeed I see a very similar memory usage. The answer is `cpu_offload`. Turn it off and you will see a dramatic change in memory usage between 1 vs many gpus.

when I turn offload off I get total gpu memory usage: i.e.: `before_init_mem_gpu+train_mem_gpu_alloc_delta+train_mem_gpu_peaked_delta` to be:
- 1 gpu: 3917MB
- 2 gpus: 2976MB (on the first gpu)

this is 25% difference on a very small model.

(same cmd as above)

(and please remember to use my PR if it's not yet merged when you try to replicate this so that `before_init_mem_gpu` metric is included)

Have a look at https://deepspeed.readthedocs.io/en/stable/memory.html
and you can quickly evaluate the needed memory with different config options turned on and the number of gpus you have. This will help you a lot to appreciate the impact of each deepspeed setting.<|||||>@stas00 Thanks for your reply!
From your experiment, I have summarized a few points:

- Use 'skip_memory_metrics 0' to report memory usage instead of 'nvidia-smi'
- As the number of GPUs increases, there is similar memory usage if 'cpu_offload' turn on.
- Memory usage will change between 1 vs many gpus if 'cpu_offload' turn off. 
- We can evaluate the needed memory through https://deepspeed.readthedocs.io/en/stable/memory.html

I still encounter some problems:

- 'https://deepspeed.readthedocs.io/en/stable/memory.html' only report the memory needed for params, optim and gradients,  This is of little value to me, because I only care about the total GPU memory during runtime
- To the paper [ZeRO-Offload](https://arxiv.org/abs/2101.06840), offload can save a lot of gpu memory, but my experiment not show that.
- Is it possible to run the gpt2-xl model (1.5B) on multiple V100 16GB GPUs? If so, how can I set up my configuration file? If not, can you tell me the reason, because I think it is feasible after reading the paper

Sincere thanks<|||||>> * 'https://deepspeed.readthedocs.io/en/stable/memory.html' only report the memory needed for params, optim and gradients,  This is of little value to me, because I only care about the total GPU memory during runtime

I hear you. I was hoping to find time to write a memory usage breakdown doc, but I'm not sure when this will happen. I have most of the information already, other than figuring out the activation memory which will require some analyzing of the model.

Would you be interesting in helping out to figure out the memory needs math for the activation memory?

> * To the paper [ZeRO-Offload](https://arxiv.org/abs/2101.06840), offload can save a lot of gpu memory, but my experiment not show that.

Best place is to ask at https://github.com/microsoft/DeepSpeed, since they wrote the paper. Probably it's the best to tag Samyam to your post there.

> * Is it possible to run the gpt2-xl model (1.5B) on multiple V100 16GB GPUs? If so, how can I set up my configuration file? If not, can you tell me the reason, because I think it is feasible after reading the paper

Should we attack this specific request in a separate issue? That's open a new issue here with the cmd and config you tried and the error (I assume OOM), your hardware spec (how many v100 16gb gpus, how much CPU memory). Ideally using HF Transformers examples so that I could reproduce it. And then we can look at finetuning it so that you can accomplish what you need. And of course please tag me.

Apologies for taking so long to follow up, been "putting out a lot of fires" recently.
<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.