[Benchmark]Why 'step' consume most time?
# 🖥 Benchmarking `transformers`

## Benchmark

I have test "**transformers/examples/pytorch/language-modeling/run_clm.py**" for "**bert-large-uncased**" with "**Deepspeed**"
but found something weird.

## Set-up

device: 4 x V100(16GB), single node

script cmd is:
```shell
#!/bin/bash
deepspeed run_mlm.py \
    --deepspeed ds_config.json \
    --model_type bert \
    --config_name bert-large-uncased \
    --output_dir output/ \
    --dataset_name wikipedia \
    --dataset_config_name 20200501.en \
    --tokenizer_name bert-large-uncased \
    --preprocessing_num_workers $(nproc) \
    --max_seq_length 128 \
    --do_train \
    --fp16 true \
    --overwrite_output_dir true \
    --per_device_train_batch_size 32
```

and the ds_config.json is:
```shell
{
    "train_micro_batch_size_per_gpu": "auto",
    "fp16": {
        "enabled": true,
        "loss_scale": 0,
        "loss_scale_window": 1000,
        "hysteresis": 2,
        "min_loss_scale": 1
    },

    "flops_profiler": {
        "enabled": true,
        "profile_step": 3,
        "module_depth": -1,
        "top_modules": 1,
        "detailed": true,
        "output_file": "./tmp.log"
      },

    "zero_optimization": {
        "stage": 1,
       "allgather_partitions": true,
       "allgather_bucket_size": 2e8,
       "reduce_scatter": true,
       "reduce_bucket_size": 2e8,
        "overlap_comm": true,
        "contiguous_gradients": true,
        "cpu_offload": true
    }
}

```

## Results

huggingface with deepspeed:
![image](https://user-images.githubusercontent.com/20511825/132789076-a00b49e3-74ca-4c04-a627-bef44dcdb76a.png)

compare to native deepspeed example "bing_bert" with same deepspeed config:
![image](https://user-images.githubusercontent.com/20511825/132791650-90e774ac-2fb8-4c19-bbf4-2cf492926536.png)


I found that native deepspeed "step" cost little time.

## question
1.why huggingface step cost most time?
2.why huggingface and deepspeed are very difference in "fwd flops per GPU" ? model are all "bert-large"
Thank you for the report, @dancingpipi.

I have a hard time making sense of it since you shared how your invoked the transformers side but not the other example. or how you measured the performance reports you shared. But I think we have enough in the numbers you shared to unravel your puzzlement.

If you look at fwd MACs (“multiply and accumulate” units) the transformers table says 1369, whereas the other example say 20, so clearly you have a ~70x difference in the amount of computation being run. And of course the performance would have a huge difference.

I suppose you're referring to this: https://github.com/microsoft/DeepSpeedExamples/tree/master/bing_bert as the other, and if you read the description it says:

> Using DeepSpeed's optimized transformer kernels as the building block, we were able to achieve the fastest BERT training record: 44 minutes on 1,024 NVIDIA V100 GPUs, compared with the previous best published result of 67 minutes on the same number and generation of GPUs.

So I hope that the "DeepSpeed's optimized transformer kernels" part answers your question. 

It's also hard to tell whether the 2 comparisons are measuring the same thing, but if they don't - CUDA kernels outperform python kernels by many times.

We are intending to look at how to gain more benefits from DeepSpeed's gifts, in particular the various kernels, we just don't have enough resources to do so fast enough. If you have the time and inspiration to write a PR that integrates such kernels into the current Bert model I'd dare to say it'd be very welcome.

BTW, pytorch core too has optimized transformers kernels that would be great to evaluate as well.

<|||||>@stas00 thanks for your great answer! 
I'm still not clearly understand why MACs are very difference between them. Is it just because of "optimized transformer kernels"? 70 times improvement is a bit unbelievable.

Another weird thing is "step" cost much time in huggingface. This is beyond my cognition.

So the current situation is that the performance of native deepspeed on bert-large is better than the performance of huggingface. 
Is there any suggestions on using huggingface with deepspeed?<|||||>> @stas00 thanks for your great answer!
> I'm still not clearly understand why MACs are very difference between them. Is it just because of "optimized transformer kernels"? 70 times improvement is a bit unbelievable.

I agree. That's why I'm suspicious of your reported numbers as 2 orders of magnitude is too fantastic to believe.

> Another weird thing is "step" cost much time in huggingface. This is beyond my cognition.

`step` is comprised of those MACs, so if your setup is not comparing apples to apples then obviously the speed difference would be out of proportion as well.
 
> So the current situation is that the performance of native deepspeed on bert-large is better than the performance of huggingface.

I haven't tested this myself, but it's almost certainly the case if one uses custom CUDA kernels vs. vanila implementation.

> Is there any suggestions on using huggingface with deepspeed?

Deepspeed is a framework/toolkit with many tools and features in it. HF transformers has currently only integrated Deepspeed's ZeRO features, which allow you to easily scale any huge model with multiple gpus and some CPU/NVMe memory and minimal to no changes to the model's code base.

Plugging Deepspeed's custom kernels has been briefly discussed as a great feature to add, but currently no one had the time to integrate it. If you have the know-how you are very welcome to spearhead the effort. I'd reach out then to the Deepspeed team via their Issues and ask whether they can help you integrate the custom cuda kernels into HF transformers. It's possible that they have already done it but the work isn't public yet. The other approach is to perhaps opening this request to the community at large and perhaps someone would be interested to work on that.
<|||||>ok, I see. Maybe I have to make a trade-off between performance and ease of use. But the first thing I want to do now is to find out why the MACs is so different.

@stas00 Thanks again for your help~<|||||>![image](https://user-images.githubusercontent.com/20511825/133065240-2e70d32e-5f9f-4f9a-bcbf-ae681d5d7624.png)

@stas00 I seem to have found the reason for the big difference in MACs, picture above is profile of "https://github.com/microsoft/DeepSpeedExamples/tree/master/bing_bert", the MACs is all zero. 

While turn-off "**deepspeed_transformer_kernel**", result looks like correct:
![image](https://user-images.githubusercontent.com/20511825/133077674-2f171f37-d992-4384-bd91-7983e5772a1b.png)

This is close to HF deepspeed's MACs.

But native deepspeed's(**without deepspeed_transformer_kernel**) "**step latency**" only cost "**58.65 us**" while HF deepspeed's cost "**96.47 ms**":
![image](https://user-images.githubusercontent.com/20511825/133081911-0bce38e7-923e-4afc-8079-b349e9adf1bf.png)

Now problem is:
1. HF deepspeed's "backward latency" is  almost twice as much as native deepspeed.
2. HF deepspeed's "step latency" cost too much time.
<|||||>I still have no idea where you get your reports from. Perhaps the first step is for you to help us reproduce your reports.<|||||>what I have compared is **https://github.com/microsoft/DeepSpeedExamples/tree/master/bing_bert** and **https://github.com/huggingface/transformers/blob/master/examples/pytorch/language-modeling/run_mlm.py**

To "**bing_bert**",  the **launch script** is: 
```shell
#!/bin/bash

base_dir=`pwd`

JOB_NAME=lamb_nvidia_data_64k_seq128
OUTPUT_DIR=${base_dir}/bert_model_nvidia_data_outputs

mkdir -p $OUTPUT_DIR

NCCL_TREE_THRESHOLD=0 deepspeed ${base_dir}/deepspeed_train.py \
--cf ${base_dir}/bert_large_lamb_nvidia_data.json \
--max_seq_length 128 \
--output_dir $OUTPUT_DIR \
--deepspeed \
--print_steps 100 \
--lr_schedule "EE" \
--lr_offset 10e-4 \
--job_name $JOB_NAME \
--deepspeed_config ${base_dir}/deepspeed_bsz64k_lamb_config_seq128.json \
--data_path_prefix ${base_dir} \
--use_nvidia_dataset
```
and "**deepspeed_bsz64k_lamb_config_seq128**.json" is :
```json
{
  "train_batch_size": 128,
  "train_micro_batch_size_per_gpu": 32,
  "steps_per_print": 100,
  "prescale_gradients": false,
  "optimizer": {
    "type": "Adam",
    "params": {
      "lr": 6e-3,
      "betas": [
        0.9,
        0.99
      ],
      "eps": 1e-8,
      "weight_decay": 0.01
    }
  },

  "flops_profiler": {
    "enabled": true,
    "profile_step": 5,
    "module_depth": -1,
    "top_modules": 1,
    "detailed": true,
    "output_file": "./tmp.log"
  },

  "zero_optimization": {
    "stage": 1,
    "overlap_comm": true,
    "allgather_partitions": true,
    "allgather_bucket_size": 5e8,
    "reduce_scatter": true,
    "reduce_bucket_size": 5e8,
    "contiguous_gradients": true,
    "grad_hooks": true,
    "round_robin_gradients": false
  },
  "scheduler": {
    "type": "WarmupLR",
    "params": {
        "warmup_min_lr": 1e-8,
        "warmup_max_lr": 6e-3
    }
  },
  "gradient_clipping": 1.0,

  "wall_clock_breakdown": false,

  "fp16": {
    "enabled": true,
    "loss_scale": 0
  },
  "sparse_attention": {
    "mode": "fixed",
    "block": 16,
    "different_layout_per_head": true,
    "num_local_blocks": 4,
    "num_global_blocks": 1,
    "attention": "bidirectional",
    "horizontal_global_attention": false,
    "num_different_global_patterns": 4
  }
}
```

To huggingface "run_mlm.py", the **launch script** is:

```shell
#!/bin/bash
deepspeed \
    run_mlm.py \
    --deepspeed ds_config.json \
    --model_type bert \
    --config_name bert-large-uncased \
    --output_dir output/ \
    --dataset_name wikipedia \
    --dataset_config_name 20200501.en \
    --tokenizer_name bert-large-uncased \
    --preprocessing_num_workers $(nproc) \
    --max_seq_length 128 \
    --do_train \
    --fp16 true \
    --overwrite_output_dir true \
    --gradient_accumulation_steps 32 \
    --per_device_train_batch_size 32
```

the **ds_config.json** is :
```json
{
    "train_micro_batch_size_per_gpu": "auto",
    "fp16": {
        "enabled": true,
        "loss_scale": 0
    },

    "flops_profiler": {
        "enabled": true,
        "profile_step": 30,
        "module_depth": -1,
        "top_modules": 1,
        "detailed": true,
        "output_file": "./tmp.log"
      },

    "zero_optimization": {
        "stage": 3,
        "allgather_partitions": true,
        "allgather_bucket_size": 5e8,
        "reduce_scatter": true,
        "reduce_bucket_size": 5e8,
         "overlap_comm": true,
         "contiguous_gradients": true,
        "grad_hooks": true,
        "round_robin_gradients": false
    },

    "gradient_clipping": 1.0,

    "wall_clock_breakdown": false,

    "sparse_attention": {
    "mode": "fixed",
    "block": 16,
    "different_layout_per_head": true,
    "num_local_blocks": 4,
    "num_global_blocks": 1,
    "attention": "bidirectional",
    "horizontal_global_attention": false,
    "num_different_global_patterns": 4
  }
}
```

My GPU is 4 x V100(16GB).

@stas00 I am pleased to provide all the information that helps to reproduce this problem<|||||>That's much better. Thank you for the explicit config sharing, @dancingpipi 

Well, let's do the "what's different in the 2 pictures" exercise:

1. you're comparing zero1 in ds to zero3 in hf - that would be quite a difference. z1 would be much faster than z3. What happens if you use the same zero stages?
2. you are using a huge `--gradient_accumulation_steps 32` in hf and none in ds - perhaps that's a huge chunk of the 70x factor?

Perhaps I missed other differences, probably re-ordering the configs' content to match will make things easier to diff.
<|||||>@stas00 thanks for your reply, maybe I made some mistake, I'll do a check<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.