Inconsistent issue in multi gpu training single machine
### System Info

Environment from `transformers-cli env`:
```
- `transformers` version: 4.26.0
- Platform: Linux-5.15.0-1033-aws-x86_64-with-glibc2.31
- Python version: 3.10.10
- Huggingface_hub version: 0.13.3
- PyTorch version (GPU?): 2.0.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes, 8 GPUs on AWS pd4.24xlarge A100 40GB chips
- Using distributed or parallel set-up in script?: Both FSDP and DeepSpeed
```

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

See Expected behavior section, script attached with run commands but unable to provide data due to sensitivity concerns

### Expected behavior

I am doing some fine-tuning on a causal LM similar to the code in https://github.com/tatsu-lab/stanford_alpaca but with additional custom data. I notice that when I fire off the multi-gpu training on a single node using a `torchrun` comand similar to:
```
torchrun --nproc_per_node=8 --master_port=8080 train.py \
    --seed 1718 \
    --model_name_or_path "facebook/opt-6.7b" \
    --output_dir "opt-6.7b" \
    --overwrite_data True  \
    --validation_size 0.05 \
    --num_train_epochs 3 \
    --per_device_train_batch_size 4 \
    --per_device_eval_batch_size 4 \
    --gradient_accumulation_steps 8 \
    --evaluation_strategy "steps" \
    --eval_steps 200 \
    --save_strategy "steps" \
    --save_steps 200 \
    --log_level "info" \
    --logging_strategy "steps" \
    --logging_steps 1 \
    --save_total_limit 1 \
    --learning_rate 2e-5 \
    --weight_decay 0. \
    --warmup_ratio 0.03 \
    --lr_scheduler_type "cosine" \
    --fsdp "full_shard auto_wrap" \
    --fsdp_transformer_layer_cls_to_wrap 'OPTDecoderLayer' \
    --bf16 True \
    --tf32 True
```
I get a bunch of warnings indicating that the local_rank of each process is -1. After stepping through some of the code in `src/trainer.py`, it looks like only data parallel gets kicked off. As such, the behavior I notice is since I requested ```--nproc_per_node=8```, it appears that literally 8 instances of my script are getting fired off per GPU I have on my machine. I can confirm this based on looking at the 8 processes running on each GPU id from the `nvidia-smi` command. Since so many processes are starting on each GPU, I get an OOM error immediately. These OOM errors occur (since the local_rank = -1) whether I use FSDP or DeepSpeed (stage 2, 3 and both with/without CPU/disk offload). 

However, when I add the "patch" below before I kicked off the training using the `Trainer` object: 

```python
    # Update local ranks
    training_args.local_rank = int(os.environ["LOCAL_RANK"])
    assert training_args.local_rank != -1, "BAD THINGS ARE ABOUT TO HAPPEN!"
    LOGGER.info(f"Configuring local ranks: I am local process: {training_args.local_rank}", main_process_only=False)
```
I finally get the single process per GPU and it looks like the GPUs are in fact doing distributed training and the script runs fine. I have attached the entire script in case this helps with debugging -- not sure what could be causing this behavior. When I run the alpaca repo, the training runs fine and no issues with the `torchrun` command. 

Any ideas if I'm doing something clearly off here? I tried upgrading to the latest version of `transformers` and the same issue. Again, no issues running the alpaca repo but mine has that local_rank = -1 problem and without the "patch" the scripts errors out right when training starts.

[train.py.zip](https://github.com/huggingface/transformers/files/11130281/train.py.zip)

Could you confirm if the issue persists with the latest release?<|||||>Hi @sgugger -- just updated to the latest version `4.27.4`. I received this error message last time about the FSDP config not being updated correctly from the command line, slipped my mind. This is why I originally rolled back to `4.26.0` based on the version that worked with the `alpaca` repo.

Either way, with the new transformers version, here are some of the error logs, still the local_rank = -1 warning and also now the `args.fsdp_config["xla"]` error, bc I think the args.fsdp_config is empty before the `Trainer` object fires off training

```
PyTorch: setting up devices
PyTorch: setting up devices
PyTorch: setting up devices
torch.distributed process group is initialized, but local_rank == -1. In order to use Torch DDP, launch your script with `python -m torch.distributed.launch
torch.distributed process group is initialized, but local_rank == -1. In order to use Torch DDP, launch your script with `python -m torch.distributed.launch
torch.distributed process group is initialized, but local_rank == -1. In order to use Torch DDP, launch your script with `python -m torch.distributed.launch
PyTorch: setting up devices
torch.distributed process group is initialized, but local_rank == -1. In order to use Torch DDP, launch your script with `python -m torch.distributed.launch
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
  File "/home/ubuntu/llm/train.py", line 585, in <module>
  File "/home/ubuntu/llm/train.py", line 585, in <module>
  File "/home/ubuntu/llm/train.py", line 585, in <module>
Traceback (most recent call last):
PyTorch: setting up devices
  File "/home/ubuntu/llm/train.py", line 585, in <module>
torch.distributed process group is initialized, but local_rank == -1. In order to use Torch DDP, launch your script with `python -m torch.distributed.launch
        train()train()    

train()  File "/home/ubuntu/llm/train.py", line 565, in train
  File "/home/ubuntu/llm/train.py", line 565, in train

  File "/home/ubuntu/llm/train.py", line 565, in train
    train()
  File "/home/ubuntu/llm/train.py", line 565, in train
    trainer = Trainer(    
trainer = Trainer(  File "/opt/conda/envs/pytorch/lib/python3.10/site-packages/transformers/trainer.py", line 421, in __init__

  File "/opt/conda/envs/pytorch/lib/python3.10/site-packages/transformers/trainer.py", line 421, in __init__
    trainer = Trainer(
  File "/opt/conda/envs/pytorch/lib/python3.10/site-packages/transformers/trainer.py", line 421, in __init__
    trainer = Trainer(
  File "/opt/conda/envs/pytorch/lib/python3.10/site-packages/transformers/trainer.py", line 421, in __init__
    if not args.fsdp_config["xla"] and args.local_rank == -1:    
if not args.fsdp_config["xla"] and args.local_rank == -1:
    if not args.fsdp_config["xla"] and args.local_rank == -1:
TypeErrorTypeError: : 'NoneType' object is not subscriptable'NoneType' object is not subscriptable

TypeError: 'NoneType' object is not subscriptable
    if not args.fsdp_config["xla"] and args.local_rank == -1:
Traceback (most recent call last):
  File "/home/ubuntu/llm/train.py", line 585, in <module>
TypeError: 'NoneType' object is not subscriptable
    train()
  File "/home/ubuntu/llm/train.py", line 565, in train
    trainer = Trainer(
  File "/opt/conda/envs/pytorch/lib/python3.10/site-packages/transformers/trainer.py", line 421, in __init__
    if not args.fsdp_config["xla"] and args.local_rank == -1:
TypeError: 'NoneType' object is not subscriptable
PyTorch: setting up devices
torch.distributed process group is initialized, but local_rank == -1. In order to use Torch DDP, launch your script with `python -m torch.distributed.launch
PyTorch: setting up devices
Traceback (most recent call last):
  File "/home/ubuntu/llm/train.py", line 585, in <module>
torch.distributed process group is initialized, but local_rank == -1. In order to use Torch DDP, launch your script with `python -m torch.distributed.launch
    train()
  File "/home/ubuntu/llm/train.py", line 565, in train
    trainer = Trainer(
  File "/opt/conda/envs/pytorch/lib/python3.10/site-packages/transformers/trainer.py", line 421, in __init__
PyTorch: setting up devices
torch.distributed process group is initialized, but local_rank == -1. In order to use Torch DDP, launch your script with `python -m torch.distributed.launch
    if not args.fsdp_config["xla"] and args.local_rank == -1:
TypeError: 'NoneType' object is not subscriptable
Traceback (most recent call last):
  File "/home/ubuntu/llm/train.py", line 585, in <module>
    train()
  File "/home/ubuntu/llm/train.py", line 565, in train
Traceback (most recent call last):
  File "/home/ubuntu/llm/train.py", line 585, in <module>
    trainer = Trainer(
  File "/opt/conda/envs/pytorch/lib/python3.10/site-packages/transformers/trainer.py", line 421, in __init__
    train()
  File "/home/ubuntu/llm/train.py", line 565, in train
    if not args.fsdp_config["xla"] and args.local_rank == -1:
TypeError:     'NoneType' object is not subscriptabletrainer = Trainer(

  File "/opt/conda/envs/pytorch/lib/python3.10/site-packages/transformers/trainer.py", line 421, in __init__
    if not args.fsdp_config["xla"] and args.local_rank == -1:
TypeError: 'NoneType' object is not subscriptable
```<|||||>cc @pacman100 <|||||>Will look into this in a few days.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.<|||||>This block of code seems to only allow FSDP with XLA? Can anyone confirm?

https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/trainer.py#L1490,L1526

When trying to follow this blog https://www.philschmid.de/sagemaker-fsdp-gpt, the entire model gets loaded onto all the gpus causing OOM although the blog tries to demonstrate FSDP (I chose an instance size with GPU mem < model size to test model sharding)
<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.