Deepspeed hang when tuning redpajama-3b
### System Info

transformers-cli says:
```
Copy-and-paste the text below in your GitHub issue and FILL OUT the two last points.

- `transformers` version: 4.30.0.dev0
- Platform: Linux-4.18.0-425.10.1.el8_7.x86_64-x86_64-with-glibc2.10
- Python version: 3.8.13
- Huggingface_hub version: 0.15.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>
```

ds_report says:
```
Copy-and-paste the text below in your GitHub issue and FILL OUT the two last points.

- `transformers` version: 4.30.0.dev0
- Platform: Linux-4.18.0-425.10.1.el8_7.x86_64-x86_64-with-glibc2.10
- Python version: 3.8.13
- Huggingface_hub version: 0.15.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

root@etc-gpu-12:/workspace# ds_report
[2023-06-07 16:33:38,748] [INFO] [real_accelerator.py:110:get_accelerator] Setting ds_accelerator to cuda (auto detect)
--------------------------------------------------
DeepSpeed C++/CUDA extension op report
--------------------------------------------------
NOTE: Ops not installed will be just-in-time (JIT) compiled at
      runtime if needed. Op compatibility means that your system
      meet the required dependencies to JIT install the op.
--------------------------------------------------
JIT compiled ops requires ninja
ninja .................. [OKAY]
--------------------------------------------------
op name ................ installed .. compatible
--------------------------------------------------
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
async_io ............... [NO] ....... [NO]
cpu_adagrad ............ [NO] ....... [OKAY]
cpu_adam ............... [NO] ....... [OKAY]
fused_adam ............. [NO] ....... [OKAY]
fused_lamb ............. [NO] ....... [OKAY]
quantizer .............. [NO] ....... [OKAY]
random_ltd ............. [NO] ....... [OKAY]
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.0
 [WARNING]  using untested triton version (2.0.0), only 1.0.0 is known to be compatible
sparse_attn ............ [NO] ....... [NO]
spatial_inference ...... [NO] ....... [OKAY]
transformer ............ [NO] ....... [OKAY]
stochastic_transformer . [NO] ....... [OKAY]
transformer_inference .. [NO] ....... [OKAY]
utils .................. [NO] ....... [OKAY]
--------------------------------------------------
DeepSpeed general environment info:
torch install path ............... ['/opt/conda/lib/python3.8/site-packages/torch']
torch version .................... 2.0.1+cu117
deepspeed install path ........... ['/opt/conda/lib/python3.8/site-packages/deepspeed']
deepspeed info ................... 0.9.4+f2f5f21b, f2f5f21b, master
torch cuda version ............... 11.7
torch hip version ................ None
nvcc version ..................... 11.6
deepspeed wheel compiled w. ...... torch 1.12, cuda 11.6
```

I am running this using a docker image from this dockerfile:
```
# https://docs.nvidia.com/deeplearning/frameworks/pytorch-release-notes/rel_22-04.html#rel_22-04
FROM nvcr.io/nvidia/pytorch:22.04-py3

# https://docs.nvidia.com/deeplearning/frameworks/pytorch-release-notes/rel-23-01.html
# This one OOM's on the tune-broken case
# FROM nvcr.io/nvidia/pytorch:23.01-py3

RUN git clone https://github.com/huggingface/transformers.git
RUN pip install transformers/.

RUN pip install git+https://github.com/huggingface/accelerate.git
# RUN git clone https://github.com/huggingface/accelerate.git
# RUN pip install accelerate/.

RUN pip install git+https://github.com/microsoft/DeepSpeed.git
# RUN git clone https://github.com/microsoft/DeepSpeed.git
# RUN pip install deepspeed/.

RUN pip install git+https://github.com/huggingface/peft.git
RUN pip install datasets evaluate loralib --upgrade --quiet
RUN pip install bitsandbytes rouge-score tensorboard py7zr einops py-spy
RUN pip install jupyter
RUN pip uninstall -y apex
RUN pip uninstall -y apex

# This is so we can run the translation test
RUN pip install -r transformers/examples/pytorch/translation/requirements.txt
```

### Who can help?

@pacman100

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Summary:
- I kick off the training script using deepspeed and NO configuration and it fails.  I've also tried with `ds_config_zero3.json` from the test directory and it fails too.

My script "tune.py":

```
#! /usr/bin/env python3

import transformers
from transformers import AutoTokenizer, AutoModelForCausalLM
from transformers import DataCollatorForLanguageModeling
from transformers import AutoModelForCausalLM, TrainingArguments, Trainer
from datasets import load_dataset

from accelerate import Accelerator

MIN_TRANSFORMERS_VERSION = '4.25.1'

assert transformers.__version__ >= MIN_TRANSFORMERS_VERSION, f'Please upgrade transformers to version {MIN_TRANSFORMERS_VERSION} or higher.'

accelerator = Accelerator()

# ==============================================================================
# DDP: Usually we use NCCL, so set that.
# Maybe need to use: NCCL_P2P_DISABLE=1
training_args = TrainingArguments(
    output_dir="redpajama-tuning-test",
    #evaluation_strategy="epoch",
    learning_rate=2e-5,
    weight_decay=0.01,
    per_device_train_batch_size=4,
    per_device_eval_batch_size=4,
    #log_level="debug",
    report_to="none",
    ddp_backend="nccl",
    ddp_timeout=60,
    push_to_hub=False
)

# =============================================================================

tokenizer = AutoTokenizer.from_pretrained("togethercomputer/RedPajama-INCITE-Base-3B-v1")
model = AutoModelForCausalLM.from_pretrained("togethercomputer/RedPajama-INCITE-Base-3B-v1")
model.train()
model = model.half()
model = model.cuda()

# =============================================================================

tokenizer.model_max_length=512
tokenizer.pad_token = tokenizer.eos_token

eli5 = load_dataset("eli5", split="train_asks[:5000]")
eli5 = eli5.train_test_split(test_size=0.2)
eli5 = eli5.flatten()

def preprocess_function(examples):
    return tokenizer([" ".join(x) for x in examples["answers.text"]])

with training_args.main_process_first(desc="tokenizing"):
    tokenized_eli5 = eli5.map(
        preprocess_function,
        batched=True,
        num_proc=4,
        remove_columns=eli5["train"].column_names
    )

block_size = 512
def group_texts(examples):
    concatenated_examples = {k: sum(examples[k], []) for k in examples.keys()}
    total_length = len(concatenated_examples[list(examples.keys())[0]])
    if total_length >= block_size:
        total_length = (total_length // block_size) * block_size
    result = {
        k: [t[i : i + block_size] for i in range(0, total_length, block_size)]
        for k, t in concatenated_examples.items()
    }
    result["labels"] = result["input_ids"].copy()
    return result

with training_args.main_process_first(desc="grouping"):
    lm_dataset = tokenized_eli5.map(group_texts, batched=True, num_proc=4)

# =================================================================================
data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=False)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=lm_dataset["train"],
    eval_dataset=lm_dataset["test"],
    tokenizer=tokenizer,
    data_collator=data_collator
)
trainer.train()
```

I run it with:
```
deepspeed tune.py
```

When it deadlock (pretty reproducibly, sometimes it completes) I use py-spy to get the stack traces.

```
root@etc-gpu-12:/workspace# py-spy dump --pid 4282
Process 4282: /opt/conda/bin/python3.8 -u tune-broken.py --local_rank=0
Python v3.8.13 (/opt/conda/bin/python3.8)

Thread 4282 (active+gil): "MainThread"
    store_flos (transformers/trainer.py:2938)
    _inner_training_loop (transformers/trainer.py:2059)
    train (transformers/trainer.py:1643)
    <module> (tune-broken.py:89)
Thread 4754 (idle): "Thread-4"
    wait (threading.py:306)
    wait (threading.py:558)
    run (tqdm/_monitor.py:60)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 4964 (idle)
Thread 4965 (idle)
root@etc-gpu-12:/workspace# py-spy dump --pid 4283
Process 4283: /opt/conda/bin/python3.8 -u tune-broken.py --local_rank=1
Python v3.8.13 (/opt/conda/bin/python3.8)

Thread 4283 (active): "MainThread"
    forward (transformers/models/gpt_neox/modeling_gpt_neox.py:278)
    _call_impl (torch/nn/modules/module.py:1501)
    forward (transformers/models/gpt_neox/modeling_gpt_neox.py:149)
    _call_impl (torch/nn/modules/module.py:1501)
    forward (transformers/models/gpt_neox/modeling_gpt_neox.py:331)
    _call_impl (torch/nn/modules/module.py:1501)
    forward (transformers/models/gpt_neox/modeling_gpt_neox.py:564)
    _call_impl (torch/nn/modules/module.py:1501)
    forward (transformers/models/gpt_neox/modeling_gpt_neox.py:673)
    _call_impl (torch/nn/modules/module.py:1501)
    _run_ddp_forward (torch/nn/parallel/distributed.py:1110)
    forward (torch/nn/parallel/distributed.py:1156)
    _call_impl (torch/nn/modules/module.py:1501)
    compute_loss (transformers/trainer.py:2763)
    training_step (transformers/trainer.py:2738)
    _inner_training_loop (transformers/trainer.py:1928)
    train (transformers/trainer.py:1643)
    <module> (tune-broken.py:89)
Thread 4824 (idle): "Thread-4"
    wait (threading.py:306)
    wait (threading.py:558)
    run (tqdm/_monitor.py:60)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 4966 (idle)
Thread 4967 (idle)
``` 

### Expected behavior

That it shouldn't deadlock.
cc @stas00 <|||||>Hello,

```
accelerator = Accelerator()
```
This shouldn't be there as Trainer creates it internally now.

Without any deepspeed config, I'm confused why you are trying to use deepspeed launcher via `deepspeed tune.py`? 

<|||||>Please refer here https://huggingface.co/docs/transformers/main_classes/deepspeed for properly using deepspeed as well as this PR #23236 in case you want to use accelerate launcher for the same.<|||||>I've been trying to follow the docs in the link above. That's why I added the issue last week about the tuning not working as described in the page. This week I am trying to get my own code working since I know that deepspeed by itself works.

Even with using the config from the test area "tests/deepspeed/ds_config_zero3.json" I get the same deadlock. If one doesn't use the config, what are all the defaults btw? I found it hard to get a clear understanding for comparison of the all the defaults.

So I guess one of my assumptions is that even if I don't take advantage of all the features and get better acceleration, deepspeed should just "work" without deadlocking?  If I do the same command "deepspeed tune.py" on one gpu it completes fine. Is that not true? My plan is to get a working baseline then tweak from there.
<|||||>Okay, after reading much further down beyond the working with multiple gpu examples I have found the "Shared Configuration" section with statements like: "be very careful that your the [Trainer](https://huggingface.co/docs/transformers/v4.29.1/en/main_classes/trainer#transformers.Trainer) arguments and DeepSpeed configurations agree. For example, are you using the same learning rate, or batch size, or gradient accumulation settings? if these mismatch the training may fail in very difficult to detect ways. You have been warned."

I will dig through the configuration files and try to debug these issues.

Any guidance on how to detect this mismatches? Is there any way to print/compare all the configs? Any help is appreciated.<|||||>Okay, now that I look at the example more closely (run_translation.py) I see that it does all the handling of args, etc. Is there a tutorial on writing a deepspeed enabled script?

Anyway, I am now int the process of passing in the deepspeed myself to my trainer using the argument properly, I still have a hang but elsewhere. It still hangs.

The stack trace is attached.
The deepspeed config is attached.
The training_args are attached.

As far as I can tell these values should work together, but I have some trouble reconciling things like "linear" and "WarmupLR".

Thoughts on debugging approaches/options?
[ds_config_zero3.txt](https://github.com/huggingface/transformers/files/11710676/ds_config_zero3.txt)
[stack_trace.txt](https://github.com/huggingface/transformers/files/11710677/stack_trace.txt)
[training_args.txt](https://github.com/huggingface/transformers/files/11710678/training_args.txt)
<|||||>Any thoughts on this?<|||||>Code: Note that I have removed few lines as they were incorrect/unnecessary

```diff
import transformers
from transformers import AutoTokenizer, AutoModelForCausalLM
from transformers import DataCollatorForLanguageModeling
from transformers import AutoModelForCausalLM, TrainingArguments, Trainer
from datasets import load_dataset

- from accelerate import Accelerator

MIN_TRANSFORMERS_VERSION = '4.25.1'

assert transformers.__version__ >= MIN_TRANSFORMERS_VERSION, f'Please upgrade transformers to version {MIN_TRANSFORMERS_VERSION} or higher.'

- accelerator = Accelerator()

# ==============================================================================
# DDP: Usually we use NCCL, so set that.
# Maybe need to use: NCCL_P2P_DISABLE=1
training_args = TrainingArguments(
    output_dir="redpajama-tuning-test",
    #evaluation_strategy="epoch",
    learning_rate=2e-5,
    weight_decay=0.01,
    per_device_train_batch_size=4,
    per_device_eval_batch_size=4,
    #log_level="debug",
    report_to="none",
    ddp_backend="nccl",
    ddp_timeout=60,
    push_to_hub=False,
+ deepspeed="ds_config_zero3.json"
)

# =============================================================================

tokenizer = AutoTokenizer.from_pretrained("togethercomputer/RedPajama-INCITE-Base-3B-v1")
model = AutoModelForCausalLM.from_pretrained("togethercomputer/RedPajama-INCITE-Base-3B-v1")
model.train()
- model = model.half()
- model = model.cuda()


# =============================================================================

tokenizer.model_max_length=512
tokenizer.pad_token = tokenizer.eos_token

eli5 = load_dataset("eli5", split="train_asks[:5000]")
eli5 = eli5.train_test_split(test_size=0.2)
eli5 = eli5.flatten()

def preprocess_function(examples):
    return tokenizer([" ".join(x) for x in examples["answers.text"]])

with training_args.main_process_first(desc="tokenizing"):
    tokenized_eli5 = eli5.map(
        preprocess_function,
        batched=True,
        num_proc=4,
        remove_columns=eli5["train"].column_names
    )

block_size = 512
def group_texts(examples):
    concatenated_examples = {k: sum(examples[k], []) for k in examples.keys()}
    total_length = len(concatenated_examples[list(examples.keys())[0]])
    if total_length >= block_size:
        total_length = (total_length // block_size) * block_size
    result = {
        k: [t[i : i + block_size] for i in range(0, total_length, block_size)]
        for k, t in concatenated_examples.items()
    }
    result["labels"] = result["input_ids"].copy()
    return result

with training_args.main_process_first(desc="grouping"):
    lm_dataset = tokenized_eli5.map(group_texts, batched=True, num_proc=4)

# =================================================================================
data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=False)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=lm_dataset["train"],
    eval_dataset=lm_dataset["test"],
    tokenizer=tokenizer,
    data_collator=data_collator
)
trainer.train()
```

ds config:
```
{
    "fp16": {
        "enabled": "auto",
        "loss_scale": 0,
        "loss_scale_window": 1000,
        "initial_scale_power": 16,
        "hysteresis": 2,
        "min_loss_scale": 1
    },

    "bf16": {
        "enabled": "auto"
    },

    "optimizer": {
        "type": "AdamW",
        "params": {
            "lr": "auto",
            "betas": "auto",
            "eps": "auto",
            "weight_decay": "auto"
        }
    },

    "scheduler": {
        "type": "WarmupLR",
        "params": {
            "warmup_min_lr": "auto",
            "warmup_max_lr": "auto",
            "warmup_num_steps": "auto"
        }
    },

    "zero_optimization": {
        "stage": 3,
        "offload_optimizer": {
            "device": "cpu",
            "pin_memory": true
        },
        "offload_param": {
            "device": "cpu",
            "pin_memory": true
        },
        "overlap_comm": true,
        "contiguous_gradients": true,
        "sub_group_size": 1e9,
        "reduce_bucket_size": "auto",
        "stage3_prefetch_bucket_size": "auto",
        "stage3_param_persistence_threshold": "auto",
        "stage3_max_live_parameters": 1e9,
        "stage3_max_reuse_distance": 1e9,
        "stage3_gather_16bit_weights_on_model_save": true
    },

    "gradient_accumulation_steps": "auto",
    "gradient_clipping": "auto",
    "steps_per_print": 2000,
    "train_batch_size": "auto",
    "train_micro_batch_size_per_gpu": "auto",
    "wall_clock_breakdown": false
}

```

command:
```
CUDA_VISIBLE_DEVICES=2,3  deepspeed issue_24090.py
```

Output logs:
```
[2023-06-22 10:30:06,594] [INFO] [real_accelerator.py:110:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2023-06-22 10:30:08,119] [WARNING] [runner.py:196:fetch_hostfile] Unable to find hostfile, will proceed with training with local resources only.
Detected CUDA_VISIBLE_DEVICES=2,3: setting --include=localhost:2,3
[2023-06-22 10:30:08,159] [INFO] [runner.py:555:main] cmd = /home/sourab/miniconda3/envs/ml/bin/python -u -m deepspeed.launcher.launch --world_info=eyJsb2NhbGhvc3QiOiBbMiwgM119 --master_addr=127.0.0.1 --master_port=29500 --enable_each_rank_log=None issue_24090.py
[2023-06-22 10:30:09,458] [INFO] [real_accelerator.py:110:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2023-06-22 10:30:10,887] [INFO] [launch.py:145:main] WORLD INFO DICT: {'localhost': [2, 3]}
[2023-06-22 10:30:10,887] [INFO] [launch.py:151:main] nnodes=1, num_local_procs=2, node_rank=0
[2023-06-22 10:30:10,887] [INFO] [launch.py:162:main] global_rank_mapping=defaultdict(<class 'list'>, {'localhost': [0, 1]})
[2023-06-22 10:30:10,887] [INFO] [launch.py:163:main] dist_world_size=2
[2023-06-22 10:30:10,887] [INFO] [launch.py:165:main] Setting CUDA_VISIBLE_DEVICES=2,3
[2023-06-22 10:30:13,111] [INFO] [real_accelerator.py:110:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2023-06-22 10:30:13,133] [INFO] [real_accelerator.py:110:get_accelerator] Setting ds_accelerator to cuda (auto detect)

===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please run

python -m bitsandbytes

 and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
================================================================================
bin /home/sourab/miniconda3/envs/ml/lib/python3.11/site-packages/bitsandbytes/libbitsandbytes_cuda118.so
/home/sourab/miniconda3/envs/ml/lib/python3.11/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: Found duplicate ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] files: {PosixPath('/home/sourab/miniconda3/envs/ml/lib/libcudart.so.11.0'), PosixPath('/home/sourab/miniconda3/envs/ml/lib/libcudart.so')}.. We'll flip a coin and try one of these, in order to fail forward.
Either way, this might cause trouble in the future:
If you get `CUDA error: invalid device function` errors, the above might be the cause and the solution is to make sure only one ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] in the paths that we search based on your env.
  warn(msg)
CUDA SETUP: CUDA runtime path found: /home/sourab/miniconda3/envs/ml/lib/libcudart.so.11.0
CUDA SETUP: Highest compute capability among GPUs detected: 8.0
CUDA SETUP: Detected CUDA version 118
CUDA SETUP: Loading binary /home/sourab/miniconda3/envs/ml/lib/python3.11/site-packages/bitsandbytes/libbitsandbytes_cuda118.so...

===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please run

python -m bitsandbytes

 and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
================================================================================
bin /home/sourab/miniconda3/envs/ml/lib/python3.11/site-packages/bitsandbytes/libbitsandbytes_cuda118.so
/home/sourab/miniconda3/envs/ml/lib/python3.11/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: Found duplicate ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] files: {PosixPath('/home/sourab/miniconda3/envs/ml/lib/libcudart.so.11.0'), PosixPath('/home/sourab/miniconda3/envs/ml/lib/libcudart.so')}.. We'll flip a coin and try one of these, in order to fail forward.
Either way, this might cause trouble in the future:
If you get `CUDA error: invalid device function` errors, the above might be the cause and the solution is to make sure only one ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] in the paths that we search based on your env.
  warn(msg)
CUDA SETUP: CUDA runtime path found: /home/sourab/miniconda3/envs/ml/lib/libcudart.so.11.0
CUDA SETUP: Highest compute capability among GPUs detected: 8.0
CUDA SETUP: Detected CUDA version 118
CUDA SETUP: Loading binary /home/sourab/miniconda3/envs/ml/lib/python3.11/site-packages/bitsandbytes/libbitsandbytes_cuda118.so...
[2023-06-22 10:30:14,921] [WARNING] [comm.py:152:init_deepspeed_backend] NCCL backend in DeepSpeed not yet implemented
[2023-06-22 10:30:14,921] [INFO] [comm.py:594:init_distributed] cdb=None
[2023-06-22 10:30:14,921] [INFO] [comm.py:625:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl
[2023-06-22 10:30:14,932] [WARNING] [comm.py:152:init_deepspeed_backend] NCCL backend in DeepSpeed not yet implemented
[2023-06-22 10:30:14,932] [INFO] [comm.py:594:init_distributed] cdb=None
[2023-06-22 10:30:23,033] [INFO] [partition_parameters.py:453:__exit__] finished initializing model with 2.78B parameters
Found cached dataset eli5 (/raid/sourab/.cache/huggingface/datasets/eli5/LFQA_reddit/1.0.0/17574e5502a10f41bbd17beba83e22475b499fa62caa1384a3d093fc856fe6fa)
Found cached dataset eli5 (/raid/sourab/.cache/huggingface/datasets/eli5/LFQA_reddit/1.0.0/17574e5502a10f41bbd17beba83e22475b499fa62caa1384a3d093fc856fe6fa)
Map (num_proc=4):   0%|                                                                        | 0/4000 [00:00<?, ? examples/s]Token indices sequence length is longer than the specified maximum sequence length for this model (661 > 512). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (1066 > 512). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (557 > 512). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (560 > 512). Running this sequence through the model will result in indexing errors
Map (num_proc=4):   0%|                                                                        | 0/1000 [00:00<?, ? examples/s]Token indices sequence length is longer than the specified maximum sequence length for this model (990 > 512). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (814 > 512). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (723 > 512). Running this sequence through the model will result in indexing errors
Map (num_proc=4):  25%|███████████████                                             | 250/1000 [00:00<00:00, 1105.17 examples/s]Token indices sequence length is longer than the specified maximum sequence length for this model (661 > 512). Running this sequence through the model will result in indexing errors
Map (num_proc=4):   0%|                                                                        | 0/4000 [00:00<?, ? examples/s]Token indices sequence length is longer than the specified maximum sequence length for this model (550 > 512). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (632 > 512). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (3324 > 512). Running this sequence through the model will result in indexing errors
Map (num_proc=4):  25%|██████████████▊                                            | 1000/4000 [00:00<00:02, 1384.05 examples/s]Token indices sequence length is longer than the specified maximum sequence length for this model (631 > 512). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (556 > 512). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (845 > 512). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (866 > 512). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (816 > 512). Running this sequence through the model will result in indexing errors
Using /raid/sourab/.cache/huggingface/torch_extensions/py311_cu118 as PyTorch extensions root...                               
Detected CUDA files, patching ldflags
Emitting ninja build file /raid/sourab/.cache/huggingface/torch_extensions/py311_cu118/cpu_adam/build.ninja...
Building extension module cpu_adam...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module cpu_adam...
Time to load cpu_adam op: 2.314084529876709 seconds
Using /raid/sourab/.cache/huggingface/torch_extensions/py311_cu118 as PyTorch extensions root...
Emitting ninja build file /raid/sourab/.cache/huggingface/torch_extensions/py311_cu118/utils/build.ninja...
Building extension module utils...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module utils...
Time to load utils op: 0.05933046340942383 seconds
Parameter Offload: Total persistent parameters: 1070080 in 258 params
Using /raid/sourab/.cache/huggingface/torch_extensions/py311_cu118 as PyTorch extensions root...
Detected CUDA files, patching ldflags
Emitting ninja build file /raid/sourab/.cache/huggingface/torch_extensions/py311_cu118/cpu_adam/build.ninja...
Building extension module cpu_adam...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module cpu_adam...
Time to load cpu_adam op: 2.3131399154663086 seconds
Using /raid/sourab/.cache/huggingface/torch_extensions/py311_cu118 as PyTorch extensions root...
Emitting ninja build file /raid/sourab/.cache/huggingface/torch_extensions/py311_cu118/utils/build.ninja...
Building extension module utils...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module utils...
Time to load utils op: 0.05986285209655762 seconds
Using /raid/sourab/.cache/huggingface/torch_extensions/py311_cu118 as PyTorch extensions root...
No modifications detected for re-loaded extension module utils, skipping build step...
Loading extension module utils...
Time to load utils op: 0.0002925395965576172 seconds
Using /raid/sourab/.cache/huggingface/torch_extensions/py311_cu118 as PyTorch extensions root...
No modifications detected for re-loaded extension module utils, skipping build step...
Loading extension module utils...
Time to load utils op: 0.00027251243591308594 seconds
  0%|                                                                                                  | 0/837 [00:00<?, ?it/s]You're using a GPTNeoXTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
You're using a GPTNeoXTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
/home/sourab/miniconda3/envs/ml/lib/python3.11/site-packages/deepspeed/runtime/zero/stage3.py:1209: UserWarning: The torch.cuda.*DtypeTensor constructors are no longer recommended. It's best to use methods such as torch.tensor(data, dtype=*, device='cuda') to create tensors. (Triggered internally at /opt/conda/conda-bld/pytorch_1687280020902/work/torch/csrc/tensor/python_tensor.cpp:83.)
  total_norm_cuda = get_accelerator().FloatTensor([float(total_norm)])
/home/sourab/miniconda3/envs/ml/lib/python3.11/site-packages/deepspeed/runtime/zero/stage3.py:1209: UserWarning: The torch.cuda.*DtypeTensor constructors are no longer recommended. It's best to use methods such as torch.tensor(data, dtype=*, device='cuda') to create tensors. (Triggered internally at /opt/conda/conda-bld/pytorch_1687280020902/work/torch/csrc/tensor/python_tensor.cpp:83.)
  total_norm_cuda = get_accelerator().FloatTensor([float(total_norm)])
  1%|█▏                                                                                     | 12/837 [01:05<1:14:36,  5.43s/it]
```

versions:
```
- `Accelerate` version: 0.21.0.dev0  -> install from main branch
- `Transformers` version: 4.31.0.dev0 -> install from main branch
- DeepSpeed version: 0.9.4
- Platform: Linux-5.4.0-125-generic-x86_64-with-glibc2.31
- Python version: 3.11.3
- Numpy version: 1.24.3
- PyTorch version (GPU?): 2.1.0.dev20230620 (True)
- PyTorch XPU available: False
- System RAM: 503.55 GB
- GPU type: NVIDIA A100-SXM4-80GB
```

Summary:
The code runs and no deadlocks or hangs. However, the warnings related to the data prep during tokenization seem concerning (not relevant to DeepSpeed/trainer)
<|||||>Thanks for looking in this!

I made the changes and tried it again.

If I take out the model.half() and model.cuda() it won't fit on my 2x 40G A100's. I tried turning on fp16 via deepspeed by setting it to true and setting `fp16=true` in my TrainingConfigs.  It starts up, but still hangs at the very end of the pass.

```
100%|██| 417/417 [15:45<00:00,  2.21s/it]
```

When I use pyspy again it is still hung up on "to" lines and down in gpt_neo.

Can you run it with fp16 turned on?

Thanks again!

EDIT: Missing "hangs" at the end of the run.<|||||>I am trying to come up with a similar version using run_clm and I can't get it to fit into memory on my gpu at all. When I turn on fp16 and model.cuda() (even without deepspeed) I can't get it to run right.

I am taking this code from a notebook (which runs fine on an a100) and trying to adapt it to run on the huggingface and deepspeed infrastructe. This seems surprising hard. The script I have above doesn't do that much at all. Why is it so hard to turn into run_clm? Is there a tutorial for this sort of thing?<|||||>Hello, what is the issue you are facing and please provide a minimal example for deep dive. The above script works with deepSpeed as I have mentioned with all the steps in detail in previous message<|||||>also could you try running with gradient checkpointing enabled so that you could fit very long sequences while training CLM models with large models. Add  `--gradient_checkpointing` to the command you run<|||||>> Hello, what is the issue you are facing and please provide a minimal example for deep dive. The above script works with deepSpeed as I have mentioned with all the steps in detail in previous message

As I mentioned before I only have 40G A100's whereas you have 80's so removing the model.half() makes it OOM for me. I tried turning on fp16 at trainer arg and the deepspeed settings but it still hangs there. What happens when you turn on fp16?

For me I have a very clear reproducible case with the fp16 parts turned on. I've attached updated versions (take off the .txt) of the ds config and the script to run with "deepspeed tune-broken.py" that should work.

[tune-broken.py.txt](https://github.com/huggingface/transformers/files/11904199/tune-broken.py.txt)
[ds_config_zero3.json.txt](https://github.com/huggingface/transformers/files/11904206/ds_config_zero3.json.txt)

> also could you try running with gradient checkpointing enabled so that you could fit very long sequences while training CLM models with large models. Add `--gradient_checkpointing` to the command you run

I'll also try the gradient checkpointing as I am down to sequence len of 400.
<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.<|||||>Hi, have you had a chance to reproduce with fp16?<|||||>Hello, changed the code to use `fp16` by adding the below line:
```diff
training_args = TrainingArguments(
    output_dir="redpajama-tuning-test",
    #evaluation_strategy="epoch",
    learning_rate=2e-5,
    weight_decay=0.01,
    per_device_train_batch_size=4,
    per_device_eval_batch_size=4,
    #log_level="debug",
    report_to="none",
    ddp_backend="nccl",
    ddp_timeout=60,
    push_to_hub=False,
+ deepspeed="ds_config_zero3.json",
+ fp16=True,
)
```

Output:
```
[2023-07-24 19:54:43,521] [INFO] [real_accelerator.py:133:get_accelerator] Setting ds_accelerator to cuda (auto detect)
The following values were not passed to `accelerate launch` and had defaults used instead:
		More than one GPU was found, enabling multi-GPU training.
		If this was unintended please pass in `--num_processes=1`.
	`--num_machines` was set to a value of `1`
	`--mixed_precision` was set to a value of `'no'`
	`--dynamo_backend` was set to a value of `'no'`
To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
[2023-07-24 19:54:48,144] [INFO] [real_accelerator.py:133:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2023-07-24 19:54:48,151] [INFO] [real_accelerator.py:133:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2023-07-24 19:54:49,459] [WARNING] [comm.py:152:init_deepspeed_backend] NCCL backend in DeepSpeed not yet implemented
[2023-07-24 19:54:49,459] [INFO] [comm.py:616:init_distributed] cdb=None
[2023-07-24 19:54:49,459] [WARNING] [comm.py:152:init_deepspeed_backend] NCCL backend in DeepSpeed not yet implemented
[2023-07-24 19:54:49,459] [INFO] [comm.py:616:init_distributed] cdb=None
[2023-07-24 19:54:49,459] [INFO] [comm.py:643:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl
[2023-07-24 19:54:59,209] [INFO] [partition_parameters.py:326:__exit__] finished initializing model with 2.78B parameters
Downloading builder script: 100%|█████████████████████████████████████████████████████████| 18.2k/18.2k [00:00<00:00, 65.8MB/s]
Downloading metadata: 100%|███████████████████████████████████████████████████████████████| 6.36k/6.36k [00:00<00:00, 38.6MB/s]
Downloading readme: 100%|█████████████████████████████████████████████████████████████████| 15.8k/15.8k [00:00<00:00, 69.3MB/s]
Found cached dataset eli5 (/raid/sourab/.cache/huggingface/datasets/eli5/LFQA_reddit/1.0.0/17574e5502a10f41bbd17beba83e22475b499fa62caa1384a3d093fc856fe6fa)
Found cached dataset eli5 (/raid/sourab/.cache/huggingface/datasets/eli5/LFQA_reddit/1.0.0/17574e5502a10f41bbd17beba83e22475b499fa62caa1384a3d093fc856fe6fa)
Map (num_proc=4):   0%|                                                                        | 0/4000 [00:00<?, ? examples/s]Token indices sequence length is longer than the specified maximum sequence length for this model (803 > 512). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (743 > 512). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (923 > 512). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (1463 > 512). Running this sequence through the model will result in indexing errors
Map (num_proc=4):   0%|                                                                        | 0/1000 [00:00<?, ? examples/s]Token indices sequence length is longer than the specified maximum sequence length for this model (770 > 512). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (608 > 512). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (810 > 512). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (516 > 512). Running this sequence through the model will result in indexing errors
Map (num_proc=4):   0%|                                                                        | 0/4000 [00:00<?, ? examples/s]Token indices sequence length is longer than the specified maximum sequence length for this model (544 > 512). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (833 > 512). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (596 > 512). Running this sequence through the model will result in indexing errors
Map (num_proc=4):  25%|██████████████▊                                            | 1000/4000 [00:00<00:02, 1432.28 examples/s]Token indices sequence length is longer than the specified maximum sequence length for this model (743 > 512). Running this sequence through the model will result in indexing errors
Map (num_proc=4):  25%|███████████████                                             | 1000/4000 [00:01<00:03, 925.60 examples/s]Token indices sequence length is longer than the specified maximum sequence length for this model (616 > 512). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (551 > 512). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (774 > 512). Running this sequence through the model will result in indexing errors
Map (num_proc=4):  25%|███████████████                                             | 250/1000 [00:00<00:00, 1145.15 examples/s]Token indices sequence length is longer than the specified maximum sequence length for this model (617 > 512). Running this sequence through the model will result in indexing errors
[2023-07-24 19:55:10,594] [WARNING] [cpu_adam.py:84:__init__] FP16 params for CPUAdam may not work on AMD CPUs                 
[2023-07-24 19:55:10,604] [WARNING] [cpu_adam.py:84:__init__] FP16 params for CPUAdam may not work on AMD CPUs
Using /raid/sourab/.cache/huggingface/torch_extensions/py310_cu118 as PyTorch extensions root...
Using /raid/sourab/.cache/huggingface/torch_extensions/py310_cu118 as PyTorch extensions root...
Detected CUDA files, patching ldflags
Emitting ninja build file /raid/sourab/.cache/huggingface/torch_extensions/py310_cu118/cpu_adam/build.ninja...
Building extension module cpu_adam...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module cpu_adam...
Time to load cpu_adam op: 2.2673165798187256 seconds
Loading extension module cpu_adam...
Time to load cpu_adam op: 2.3539650440216064 seconds
Parameter Offload: Total persistent parameters: 1070080 in 258 params
  0%|                                                                                                  | 0/840 [00:00<?, ?it/s]You're using a GPTNeoXTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
You're using a GPTNeoXTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
  1%|▋                                                                                       | 6/840 [01:27<3:25:40, 14.80s/it]
```

Also, the memory usage is `21489MiB` (21GB vram) on both the GPUs<|||||>Can you post the final results please?