`rank = dist.get_rank()` throws `group error` while loading model with running `AutoModelForSeq2SeqLM.from_pretrained` using deepspeed
### System Info

- `transformers` version: 4.25.1
- Platform: Linux-3.10.0-514.26.2.el7.x86_64-x86_64-with-glibc2.10
- Python version: 3.8.5
- Huggingface_hub version: 0.11.1
- PyTorch version (GPU?): 1.8.0a0+1606899 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: yes(deepseed)

### Who can help?

@stas00 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I'm trying to test running `examples/pytorch/translation/run_translation.py` with `deepspeed`, using this [example](https://github.com/huggingface/transformers/issues/17534#issuecomment-1146249686) @stas00 had written (thanks beforehand)

- script I've run
```
rm -r output_dir; PYTHONPATH=src USE_TF=0  deepspeed  --num_gpus 4 \
examples/pytorch/translation/run_translation.py --model_name_or_path t5-small \
--output_dir output_dir --overwrite_output_dir --max_source_length 128 \
--max_target_length 128 --val_max_target_length 128 --do_train \
--num_train_epochs 1 --learning_rate 3e-3 \
--dataset_name wmt16 --dataset_config ro-en --source_lang en --target_lang ro \
--source_prefix 'translate English to Romanian: ' --max_train_samples 5 \
--deepspeed tests/deepspeed/ds_config_zero3_test.json --save_steps 5
```

- `ds_config_zero3_test.json`
    - I changed `gradient_accumulation_steps, train_batch_size, train_micro_batch_size_per_gpu` values from `auto` to some int values, since `auto` value threw out error such as `check batch related parameters. train_batch_size is not equal to micro_batch_per_gpu * gradient_acc_step * world_size 8 != 2 * 1 * 1`
```
%%bash
cat <<'EOT' > ds_config_zero3_test.json
{
    "fp16": {
        "enabled": "auto",
        "loss_scale": 0,
        "loss_scale_window": 1000,
        "initial_scale_power": 16,
        "hysteresis": 2,
        "min_loss_scale": 1
    },

    "optimizer": {
        "type": "AdamW",
        "params": {
            "lr": "auto",
            "betas": "auto",
            "eps": "auto",
            "weight_decay": "auto"
        }
    },

    "scheduler": {
        "type": "WarmupLR",
        "params": {
            "warmup_min_lr": "auto",
            "warmup_max_lr": "auto",
            "warmup_num_steps": "auto"
        }
    },

    "zero_optimization": {
        "stage": 3,
        "offload_optimizer": {
            "device": "cpu",
            "pin_memory": true
        },
        "offload_param": {
            "device": "cpu",
            "pin_memory": true
        },
        "overlap_comm": true,
        "contiguous_gradients": true,
        "sub_group_size": 1e9,
        "reduce_bucket_size": "auto",
        "stage3_prefetch_bucket_size": "auto",
        "stage3_param_persistence_threshold": "auto",
        "stage3_max_live_parameters": 1e9,
        "stage3_max_reuse_distance": 1e9,
        "stage3_gather_fp16_weights_on_model_save": true
    },

    "gradient_accumulation_steps": 1,
    "gradient_clipping": "auto",
    "steps_per_print": 2000,
    "train_batch_size": 2,
    "train_micro_batch_size_per_gpu": 2,
    "wall_clock_breakdown": false
}
EOT
```

- ERROR message
    - An error occurs while loading model, getting rank info using `torch.distributed.get_rank`, throwing `RuntimeError: The given group does not exist`
    - Maybe it's because I'm using older version of PyTorch? (`1.8.0a0+1606899`). 
```
[2023-01-09 00:34:14,932] [INFO] [partition_parameters.py:709:__init__] _all_gather_base API is not available in torch 1.8.0a0+1606899
Traceback (most recent call last):
  File "examples/pytorch/translation/run_translation.py", line 660, in <module>
    main()
  File "examples/pytorch/translation/run_translation.py", line 374, in main
    model = AutoModelForSeq2SeqLM.from_pretrained(
  File "/home/transformers/src/transformers/models/auto/auto_factory.py", line 463, in from_pretrained
    return model_class.from_pretrained(
  File "/home/transformers/src/transformers/modeling_utils.py", line 2299, in from_pretrained
    with ContextManagers(init_contexts):
  File "/home/transformers/src/transformers/utils/generic.py", line 359, in __enter__
    self.stack.enter_context(context_manager)
  File "/opt/conda/lib/python3.8/contextlib.py", line 425, in enter_context
    result = _cm_type.__enter__(cm)
  File "/opt/conda/lib/python3.8/site-packages/deepspeed/runtime/zero/partition_parameters.py", line 400, in __enter__
    print_rank_0(
  File "/opt/conda/lib/python3.8/site-packages/deepspeed/runtime/zero/partition_parameters.py", line 49, in print_rank_0
    rank = dist.get_rank()
  File "/opt/conda/lib/python3.8/site-packages/deepspeed/comm/comm.py", line 575, in get_rank
    return cdb.get_rank(group)
  File "/opt/conda/lib/python3.8/site-packages/deepspeed/comm/torch.py", line 175, in get_rank
    return torch.distributed.get_rank(group=group)
  File "/opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 645, in get_rank
    return _get_group_rank(group, _default_pg.rank())
  File "/opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 191, in _get_group_rank
    raise RuntimeError("The given group does not exist")
RuntimeError: The given group does not exist
```

- `ds_report`
```
--------------------------------------------------
DeepSpeed C++/CUDA extension op report
--------------------------------------------------
NOTE: Ops not installed will be just-in-time (JIT) compiled at
      runtime if needed. Op compatibility means that your system
      meet the required dependencies to JIT install the op.
--------------------------------------------------
JIT compiled ops requires ninja
ninja .................. [OKAY]
--------------------------------------------------
op name ................ installed .. compatible
--------------------------------------------------
cpu_adam ............... [NO] ....... [OKAY]
cpu_adagrad ............ [NO] ....... [OKAY]
fused_adam ............. [NO] ....... [OKAY]
fused_lamb ............. [NO] ....... [OKAY]
 [WARNING]  please install triton==1.0.0 if you want to use sparse attention
sparse_attn ............ [NO] ....... [NO]
transformer ............ [NO] ....... [OKAY]
stochastic_transformer . [NO] ....... [OKAY]
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
async_io ............... [NO] ....... [NO]
utils .................. [NO] ....... [OKAY]
quantizer .............. [NO] ....... [OKAY]
transformer_inference .. [NO] ....... [OKAY]
spatial_inference ...... [NO] ....... [OKAY]
--------------------------------------------------
DeepSpeed general environment info:
torch install path ............... ['/opt/conda/lib/python3.8/site-packages/torch']
torch version .................... 1.8.0a0+1606899
torch cuda version ............... 11.1
torch hip version ................ None
nvcc version ..................... 11.1
deepspeed install path ........... ['/opt/conda/lib/python3.8/site-packages/deepspeed']
deepspeed info ................... 0.7.7, unknown, unknown
deepspeed wheel compiled w. ...... torch 1.8, cuda 11.1
```

### Expected behavior

I've expected it to run with multiple gpus but not running
That traceback doesn't look like an issue in the HF integration,  indeed could you try some more recent pytorch first? You're not even using a released version of 1.8, but some nightly/rc version (`torch version .................... 1.8.0a0+1606899`). I'd try 1.12 or 1.13 (latest).

Please let me know if it doesn't help and I will try to reproduce it.<|||||>> That traceback doesn't look like an issue in the HF integration, indeed could you try some more recent pytorch first? You're not even using a released version of 1.8, but some nightly/rc version (`torch version .................... 1.8.0a0+1606899`). I'd try 1.12 or 1.13 (latest).
> 
> Please let me know if it doesn't help and I will try to reproduce it.

I've changed the environment to following setting
- `transformers` version: 4.22.2
- Platform: Linux-4.19.93-1.nbp.el7.x86_64-x86_64-with-glibc2.10
- Python version: 3.8.13
- Huggingface_hub version: 0.11.0
- PyTorch version (GPU?): 1.13.0a0+08820cb (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed

And I succeeded running the script:) Thank you
Although, I've noticed that when using deepspeed with huggingface Trainer, Training info gives `Number of trainable parameters` as zero
```
[INFO|trainer.py:1643] 2023-01-10 04:55:31,025 >> ***** Running training *****
[INFO|trainer.py:1644] 2023-01-10 04:55:31,025 >>   Num examples = 10
[INFO|trainer.py:1645] 2023-01-10 04:55:31,025 >>   Num Epochs = 3
[INFO|trainer.py:1646] 2023-01-10 04:55:31,025 >>   Instantaneous batch size per device = 8
[INFO|trainer.py:1647] 2023-01-10 04:55:31,025 >>   Total train batch size (w. parallel, distributed & accumulation) = 16
[INFO|trainer.py:1648] 2023-01-10 04:55:31,025 >>   Gradient Accumulation steps = 1
[INFO|trainer.py:1649] 2023-01-10 04:55:31,025 >>   Total optimization steps = 3
[INFO|trainer.py:1650] 2023-01-10 04:55:31,028 >>   Number of trainable parameters = 0
```

The same issue is here(https://discuss.huggingface.co/t/deepspeed-with-trainer-no-of-trainable-parameters-coming-to-be-0/27187).
Thank you in advance @stas00:) I really appreciate your kindness<|||||>great to hear that it worked, @SoundProvider 

> Although, I've noticed that when using deepspeed with huggingface Trainer, Training info gives Number of trainable parameters as zero

That means that the params weren't gathered under zero3. and when zero3 is used deepspeed puts placeholders with tensors of zero3. Please create a new issue and I will fix it. or if you feel inspired you can contribute a few lines of code that will check if the model is running under deepspeed and gather the params. It'd be something like this:

```
    if is_deepspeed_zero3_enabled():
        import deepspeed
        size = 0
        for param in model.parameters():
            with deepspeed.zero.GatheredParameters(param, modifier_rank=None):
                size += param.numel()
```

we do it one param at a time to avoid loading a potentially huge model onto cpu.<|||||>I'd love to try it out.
I will go through some the codes and make a new issue if I find a way:)
Thank you<|||||>Wonderful!

The other service we could provide to future users is to find out which minimal pt version is required to make it work and assert if it's not the right one - in case you're interested to explore that one - but by all means this is only an invitation, please feel no pressure to do anything unless it gives you goosebumps when you think of doing it.<|||||>@stas00 
Hello Stas. I've tested running two different models with both deepspeed and torch DDP. As you can see below, t5-large with deepspeed uses much less GPU memory than torch DDP, while OPT model with deepspeed doesn't show useful decrease.
I've looked through deepspeed codes couldn't find any hints,,
I have 2 questions
- What would cause the differenct GPU memory decrease between two models? From what I've understood from [deepspeed.initialize](https://github.com/microsoft/DeepSpeed/blob/fe728e3ed880f27de2c21234f12b7aa6f672e825/deepspeed/runtime/pipe/engine.py#L138), deepspeed handles only tensors, not model blocks.
- After what I read from[ deepspeed memory efficiency](https://www.deepspeed.ai/training/#memory-efficiency), I expected t5-large with deepspeed would show much more GPU memory decrease than I tested. Could you tell me any hint?
Thank you beforehand for your great works


### Experiments
1. t5-large
    - deepspeed
      - script: `rm -r output_dir; PYTHONPATH=src USE_TF=0  deepspeed  --num_gpus 2 \
  examples/pytorch/translation/run_translation.py --model_name_or_path t5-large \
  --output_dir output_dir --overwrite_output_dir --max_source_length 128 \
  --max_target_length 128 --val_max_target_length 128 --do_train \
  --num_train_epochs 1 --learning_rate 3e-3 \
  --dataset_name wmt16 --dataset_config ro-en --source_lang en --target_lang ro \
  --source_prefix 'translate English to Romanian: ' --max_train_samples 5 \
  --deepspeed tests/deepspeed/ds_config_zero3_NSML_test.json --per_device_train_batch_size 1`
        - ![image](https://user-images.githubusercontent.com/48939336/212818170-69ec13ab-bceb-4fbc-9099-2b7e72b08e48.png)
    - torch DDP
        - script: `python -m torch.distributed.launch --nproc_per_node=2 examples/pytorch/translation/run_translation.py --model_name_or_path t5-large \
--output_dir output_dir --overwrite_output_dir --max_source_length 128 \
--max_target_length 128 --val_max_target_length 128 --do_train \
--num_train_epochs 10 --learning_rate 3e-3 \
--dataset_name wmt16 --dataset_config ro-en --source_lang en --target_lang ro \
--source_prefix 'translate English to Romanian: ' --max_train_samples 5 --per_device_train_batch_size 1`
        - ![image](https://user-images.githubusercontent.com/48939336/212818484-ca76c498-7725-4b31-ae2c-2f25e4633462.png)

2. OPT
    - model: [link](https://github.com/huggingface/transformers/tree/main/src/transformers/models/opt), version: 4.26.0.dev0
        - used [OPTForCausalLM](https://github.com/huggingface/transformers/blob/2411f0e465e761790879e605a4256f3d4afb7f82/src/transformers/models/opt/modeling_opt.py#L808), with custom dataset
    - deepspeed
        - script: `rm -r output_dir; PYTHONPATH=src USE_TF=0  deepspeed  --num_gpus 2 \
run_opt.py --model_name_or_path facebook/opt-1.3b --output_dir test \
--deepspeed ../tests/deepspeed/ds_config_zero3_NSML_test.json --do_train True --do_eval True \
--per_device_train_batch_size 1`
        - ![image](https://user-images.githubusercontent.com/48939336/212819257-6da9ef94-21a0-4d7c-8b4a-3f759588df77.png)
    - torch DDP
        - script: `rm -r test; python -m torch.distributed.launch --nproc_per_node=2 run_opt.py --model_name_or_path facebook/opt-1.3b --output_dir test \
--do_train True --do_eval True --per_device_train_batch_size 1`
        - ![image](https://user-images.githubusercontent.com/48939336/212819377-74b3bd99-0036-4c61-8995-1156d897dfcd.png)


#### env info
- `transformers` version: 4.26.0.dev0
- Platform: Linux-4.19.93-1.nbp.el7.x86_64-x86_64-with-glibc2.10
- Python version: 3.8.13
- Huggingface_hub version: 0.11.0
- PyTorch version (GPU?): 1.13.0a0+08820cb (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
<|||||>I won't trust `nvidia-smi` for measuring memory usage patterns, as it is not aware of cuda caching and you can't see peak memory usage either.

You can repeat the above runs, but add `--skip_memory_metrics 0` and it'll print you all the memory usage stats at the end of each run. (only use this for debug as it slows training down)

I'm not saying that you still won't see an issue, but I'm asking to do that as it'd give us a much precise memory usage stats.

and ideally please make it into a new Issue and let's close this one. As this discussion is now totally unrelated to the topic of this Issue.

Thanks.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.