AttributeError: 'DeepSpeedCPUAdam' object has no attribute 'ds_opt_adam' for multinode training
### System Info

- `transformers` version: 4.31.0.dev0
- Platform: Linux-4.18.0-305.25.1.el8_4.x86_64-x86_64-with-glibc2.28
- Python version: 3.10.12
- Huggingface_hub version: 0.15.1
- Safetensors version: 0.3.1
- Accelerate version: 0.21.0
- Accelerate config:    not found
- PyTorch version (GPU?): 1.13.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: multinode distributed setup
- Deepspeed version: 0.9.5

### Who can help?

@pacman100

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

To reproduce, run the following with your `hostfile` and `deepspeed` config specified. I used the zero3 config [here](https://huggingface.co/docs/transformers/main_classes/deepspeed#zero3-config).

```
deepspeed --hostfile=$PBS_O_WORKDIR/hostfile $ROOT_DIR/transformers/examples/pytorch/language-modeling/run_clm.py \
    --model_name_or_path gpt2 \
    --dataset_name wikitext \
    --dataset_config_name wikitext-2-raw-v1 \
    --per_device_train_batch_size 8 \
    --per_device_eval_batch_size 8 \
    --do_train \
    --do_eval \
    --output_dir ~/scratch/finetuned_models/debug \
    --deepspeed "$ROOT_DIR/FastChat/fastchat/ds_config/ds_config_zero3.json"
```

I ran into the error `AttributeError: 'DeepSpeedCPUAdam' object has no attribute 'ds_opt_adam'`. Here's the logs:

```
x1000c1s1b0n0:   warn(msg)
x1000c1s1b0n0: /home/users/industry/dso/lannliat/.local/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/opt/cray/pe/cce/13.0.2/cce/aarch64/lib/libcray-c++-rts.a')}
x1000c1s1b0n0:   warn(msg)
x1000c1s1b0n0: /home/users/industry/dso/lannliat/.local/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/opt/cray/pe/cti/2.15.10/lib/pkgconfig')}
x1000c1s1b0n0:   warn(msg)
x1000c1s1b0n0: /home/users/industry/dso/lannliat/.local/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/opt/cray/pe/cce/13.0.2/cce/aarch64/lib')}
x1000c1s1b0n0:   warn(msg)
x1000c1s1b0n0: /home/users/industry/dso/lannliat/.local/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/opt/cray/pe/cce/13.0.2/cce/x86_64/share/nls/En/%N.cat')}
x1000c1s1b0n0:   warn(msg)
x1000c1s1b0n0: /home/users/industry/dso/lannliat/.local/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/opt/cray/pe/mpich/8.1.15/ofi/@PRGENV@/@PE_MPICH_GENCOMPS@/lib/pkgconfig')}
x1000c1s1b0n0:   warn(msg)
x1000c1s1b0n0: /home/users/industry/dso/lannliat/.local/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('libfabric/1.11.0.4.125'), PosixPath('cray-pals/1.1.6'), PosixPath('craype-x86-rome'), PosixPath('craype-network-ofi'), PosixPath('cce/13.0.2'), PosixPath('cray-dsmml/0.2.2'), PosixPath('perftools-base/22.04.0'), PosixPath('cray-mpich/8.1.15'), PosixPath('craype/2.7.15'), PosixPath('PrgEnv-cray/8.3.3')}
x1000c1s1b0n0:   warn(msg)
x1000c1s1b0n0: /home/users/industry/dso/lannliat/.local/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/opt/cray/pe/cce/13.0.2/cce/aarch64/include/craylibs')}
x1000c1s1b0n0:   warn(msg)
x1000c1s1b0n0: /home/users/industry/dso/lannliat/.local/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/opt/gcc-cross-aarch64/8.1.0/aarch64')}
x1000c1s1b0n0:   warn(msg)
x1000c1s1b0n0: /home/users/industry/dso/lannliat/.local/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/opt/cray/pe/fftw/3.3.8.13/@PE_FFTW_DEFAULT_TARGET@/lib/pkgconfig')}
x1000c1s1b0n0:   warn(msg)
x1000c1s1b0n0: /home/users/industry/dso/lannliat/.local/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/opt/cray/pe/libsci/21.08.1.2/@PRGENV@/@PE_LIBSCI_DEFAULT_GENCOMPS@/@PE_LIBSCI_DEFAULT_TARGET@/lib/pkgconfig')}
x1000c1s1b0n0:   warn(msg)
x1000c1s1b0n0: /home/users/industry/dso/lannliat/.local/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/opt/cray/pe/sma/11.5.3.beta/ofi/sma@PE_SMA_DEFAULT_DIR_DEFAULT64@/lib64/pkgconfig')}
x1000c1s1b0n0:   warn(msg)
x1000c1s1b0n0: CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
x1000c1s1b0n0: /home/users/industry/dso/lannliat/.local/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/opt/clmgr/man'), PosixPath('/opt/cray/pe/man')}
x1000c1s1b0n0:   warn(msg)
x1000c1s1b0n0: /home/users/industry/dso/lannliat/.local/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/opt/modulefiles')}
x1000c1s1b0n0:   warn(msg)
x1000c1s1b0n0: /home/users/industry/dso/lannliat/.local/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('libexec64/opts')}
x1000c1s1b0n0:   warn(msg)
x1000c1s1b0n0: CUDA SETUP: Highest compute capability among GPUs detected: 8.0
x1000c1s1b0n0: CUDA SETUP: Detected CUDA version 116
x1000c1s1b0n0: CUDA SETUP: Loading binary /home/users/industry/dso/lannliat/.local/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cpu.so...
x1000c1s1b0n0: /home/users/industry/dso/lannliat/.local/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('() {  eval `/opt/cray/pe/modules/3.2.11.6/bin/modulecmd bash $*`\n}')}
x1000c1s1b0n0:   warn(msg)
x1000c1s1b0n0: /home/users/industry/dso/lannliat/.local/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/usr/local/cuda/lib64')}
x1000c1s1b0n0:   warn(msg)
x1000c1s1b0n0: /home/users/industry/dso/lannliat/.local/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: No libcudart.so found! Install CUDA or the cudatoolkit package (anaconda)!
x1000c1s1b0n0:   warn(msg)
x1000c1s1b0n0: [2023-07-16 11:23:41,585] [INFO] [real_accelerator.py:110:get_accelerator] Setting ds_accelerator to cuda (auto detect)
x1000c1s1b0n0: [2023-07-16 11:23:41,586] [INFO] [real_accelerator.py:110:get_accelerator] Setting ds_accelerator to cuda (auto detect)
x1000c0s6b0n0: [2023-07-16 11:23:44,841] [WARNING] [comm.py:152:init_deepspeed_backend] NCCL backend in DeepSpeed not yet implemented
x1000c0s6b0n0: [2023-07-16 11:23:44,841] [INFO] [comm.py:594:init_distributed] cdb=None
x1000c0s6b0n0: [2023-07-16 11:23:44,841] [INFO] [comm.py:625:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl
x1000c0s6b0n0: [2023-07-16 11:23:44,842] [WARNING] [comm.py:152:init_deepspeed_backend] NCCL backend in DeepSpeed not yet implemented
x1000c0s6b0n0: [2023-07-16 11:23:44,842] [INFO] [comm.py:594:init_distributed] cdb=None
x1000c1s1b0n0: [2023-07-16 11:23:46,769] [WARNING] [comm.py:152:init_deepspeed_backend] NCCL backend in DeepSpeed not yet implemented
x1000c1s1b0n0: [2023-07-16 11:23:46,770] [WARNING] [comm.py:152:init_deepspeed_backend] NCCL backend in DeepSpeed not yet implemented
x1000c1s1b0n0: [2023-07-16 11:23:46,770] [INFO] [comm.py:594:init_distributed] cdb=None
x1000c1s1b0n0: [2023-07-16 11:23:46,770] [INFO] [comm.py:594:init_distributed] cdb=None
x1000c0s6b0n0: 07/16/2023 11:23:47 - WARNING - __main__ - Process rank: 0, device: cuda:0, n_gpu: 1distributed training: True, 16-bits training: False
x1000c0s6b0n0: 07/16/2023 11:23:47 - INFO - __main__ - Training/evaluation parameters TrainingArguments(
x1000c0s6b0n0: _n_gpu=1,
x1000c0s6b0n0: adafactor=False,
x1000c0s6b0n0: adam_beta1=0.9,
x1000c0s6b0n0: adam_beta2=0.999,
x1000c0s6b0n0: adam_epsilon=1e-08,
x1000c0s6b0n0: auto_find_batch_size=False,
x1000c0s6b0n0: bf16=False,
x1000c0s6b0n0: bf16_full_eval=False,
x1000c0s6b0n0: data_seed=None,
x1000c0s6b0n0: dataloader_drop_last=False,
x1000c0s6b0n0: dataloader_num_workers=0,
x1000c0s6b0n0: dataloader_pin_memory=True,
x1000c0s6b0n0: ddp_backend=None,
x1000c0s6b0n0: ddp_broadcast_buffers=None,
x1000c0s6b0n0: ddp_bucket_cap_mb=None,
x1000c0s6b0n0: ddp_find_unused_parameters=None,
x1000c0s6b0n0: ddp_timeout=1800,
x1000c0s6b0n0: debug=[],
x1000c0s6b0n0: deepspeed=/home/users/industry/dso/lannliat/FastChat/fastchat/ds_config/ds_config_zero3.json,
x1000c0s6b0n0: disable_tqdm=False,
x1000c0s6b0n0: do_eval=True,
x1000c0s6b0n0: do_predict=False,
x1000c0s6b0n0: do_train=True,
x1000c0s6b0n0: eval_accumulation_steps=None,
x1000c0s6b0n0: eval_delay=0,
x1000c0s6b0n0: eval_steps=None,
x1000c0s6b0n0: evaluation_strategy=no,
x1000c0s6b0n0: fp16=False,
x1000c0s6b0n0: fp16_backend=auto,
x1000c0s6b0n0: fp16_full_eval=False,
x1000c0s6b0n0: fp16_opt_level=O1,
x1000c0s6b0n0: fsdp=[],
x1000c0s6b0n0: fsdp_config={'fsdp_min_num_params': 0, 'xla': False, 'xla_fsdp_grad_ckpt': False},
x1000c0s6b0n0: fsdp_min_num_params=0,
x1000c0s6b0n0: fsdp_transformer_layer_cls_to_wrap=None,
x1000c0s6b0n0: full_determinism=False,
x1000c0s6b0n0: gradient_accumulation_steps=1,
x1000c0s6b0n0: gradient_checkpointing=False,
x1000c0s6b0n0: greater_is_better=None,
x1000c0s6b0n0: group_by_length=False,
x1000c0s6b0n0: half_precision_backend=auto,
x1000c0s6b0n0: hub_model_id=None,
x1000c0s6b0n0: hub_private_repo=False,
x1000c0s6b0n0: hub_strategy=every_save,
x1000c0s6b0n0: hub_token=<HUB_TOKEN>,
x1000c0s6b0n0: ignore_data_skip=False,
x1000c0s6b0n0: include_inputs_for_metrics=False,
x1000c0s6b0n0: jit_mode_eval=False,
x1000c0s6b0n0: label_names=None,
x1000c0s6b0n0: label_smoothing_factor=0.0,
x1000c0s6b0n0: learning_rate=5e-05,
x1000c0s6b0n0: length_column_name=length,
x1000c0s6b0n0: load_best_model_at_end=False,
x1000c0s6b0n0: local_rank=0,
x1000c0s6b0n0: log_level=passive,
x1000c0s6b0n0: log_level_replica=warning,
x1000c0s6b0n0: log_on_each_node=True,
x1000c0s6b0n0: logging_dir=/home/users/industry/dso/lannliat/scratch/finetuned_models/debug/runs/Jul16_11-23-44_x1000c0s6b0n0,
x1000c0s6b0n0: logging_first_step=False,
x1000c0s6b0n0: logging_nan_inf_filter=True,
x1000c0s6b0n0: logging_steps=500,
x1000c0s6b0n0: logging_strategy=steps,
x1000c0s6b0n0: lr_scheduler_type=linear,
x1000c0s6b0n0: max_grad_norm=1.0,
x1000c0s6b0n0: max_steps=-1,
x1000c0s6b0n0: metric_for_best_model=None,
x1000c0s6b0n0: mp_parameters=,
x1000c0s6b0n0: no_cuda=False,
x1000c0s6b0n0: num_train_epochs=3.0,
x1000c0s6b0n0: optim=adamw_hf,
x1000c0s6b0n0: optim_args=None,
x1000c0s6b0n0: output_dir=/home/users/industry/dso/lannliat/scratch/finetuned_models/debug,
x1000c0s6b0n0: overwrite_output_dir=False,
x1000c0s6b0n0: past_index=-1,
x1000c0s6b0n0: per_device_eval_batch_size=8,
x1000c0s6b0n0: per_device_train_batch_size=8,
x1000c0s6b0n0: prediction_loss_only=False,
x1000c0s6b0n0: push_to_hub=False,
x1000c0s6b0n0: push_to_hub_model_id=None,
x1000c0s6b0n0: push_to_hub_organization=None,
x1000c0s6b0n0: push_to_hub_token=<PUSH_TO_HUB_TOKEN>,
x1000c0s6b0n0: ray_scope=last,
x1000c0s6b0n0: remove_unused_columns=True,
x1000c0s6b0n0: report_to=['wandb'],
x1000c0s6b0n0: resume_from_checkpoint=None,
x1000c0s6b0n0: run_name=/home/users/industry/dso/lannliat/scratch/finetuned_models/debug,
x1000c0s6b0n0: save_on_each_node=False,
x1000c0s6b0n0: save_safetensors=False,
x1000c0s6b0n0: save_steps=500,
x1000c0s6b0n0: save_strategy=steps,
x1000c0s6b0n0: save_total_limit=None,
x1000c0s6b0n0: seed=42,
x1000c0s6b0n0: sharded_ddp=[],
x1000c0s6b0n0: skip_memory_metrics=True,
x1000c0s6b0n0: tf32=None,
x1000c0s6b0n0: torch_compile=False,
x1000c0s6b0n0: torch_compile_backend=None,
x1000c0s6b0n0: torch_compile_mode=None,
x1000c0s6b0n0: torchdynamo=None,
x1000c0s6b0n0: tpu_metrics_debug=False,
x1000c0s6b0n0: tpu_num_cores=None,
x1000c0s6b0n0: use_ipex=False,
x1000c0s6b0n0: use_legacy_prediction_loop=False,
x1000c0s6b0n0: use_mps_device=False,
x1000c0s6b0n0: warmup_ratio=0.0,
x1000c0s6b0n0: warmup_steps=0,
x1000c0s6b0n0: weight_decay=0.0,
x1000c0s6b0n0: xpu_backend=None,
x1000c0s6b0n0: )
x1000c0s6b0n0: 07/16/2023 11:23:47 - WARNING - __main__ - Process rank: 1, device: cuda:1, n_gpu: 1distributed training: True, 16-bits training: False
x1000c1s1b0n0: 07/16/2023 11:23:47 - WARNING - __main__ - Process rank: 0, device: cuda:0, n_gpu: 1distributed training: True, 16-bits training: False
x1000c1s1b0n0: 07/16/2023 11:23:47 - INFO - __main__ - Training/evaluation parameters TrainingArguments(
x1000c1s1b0n0: _n_gpu=1,
x1000c1s1b0n0: adafactor=False,
x1000c1s1b0n0: adam_beta1=0.9,
x1000c1s1b0n0: adam_beta2=0.999,
x1000c1s1b0n0: adam_epsilon=1e-08,
x1000c1s1b0n0: auto_find_batch_size=False,
x1000c1s1b0n0: bf16=False,
x1000c1s1b0n0: bf16_full_eval=False,
x1000c1s1b0n0: data_seed=None,
x1000c1s1b0n0: dataloader_drop_last=False,
x1000c1s1b0n0: dataloader_num_workers=0,
x1000c1s1b0n0: dataloader_pin_memory=True,
x1000c1s1b0n0: ddp_backend=None,
x1000c1s1b0n0: ddp_broadcast_buffers=None,
x1000c1s1b0n0: ddp_bucket_cap_mb=None,
x1000c1s1b0n0: ddp_find_unused_parameters=None,
x1000c1s1b0n0: ddp_timeout=1800,
x1000c1s1b0n0: debug=[],
x1000c1s1b0n0: deepspeed=/home/users/industry/dso/lannliat/FastChat/fastchat/ds_config/ds_config_zero3.json,
x1000c1s1b0n0: disable_tqdm=False,
x1000c1s1b0n0: do_eval=True,
x1000c1s1b0n0: do_predict=False,
x1000c1s1b0n0: do_train=True,
x1000c1s1b0n0: eval_accumulation_steps=None,
x1000c1s1b0n0: eval_delay=0,
x1000c1s1b0n0: eval_steps=None,
x1000c1s1b0n0: evaluation_strategy=no,
x1000c1s1b0n0: fp16=False,
x1000c1s1b0n0: fp16_backend=auto,
x1000c1s1b0n0: fp16_full_eval=False,
x1000c1s1b0n0: fp16_opt_level=O1,
x1000c1s1b0n0: fsdp=[],
x1000c1s1b0n0: fsdp_config={'fsdp_min_num_params': 0, 'xla': False, 'xla_fsdp_grad_ckpt': False},
x1000c1s1b0n0: fsdp_min_num_params=0,
x1000c1s1b0n0: fsdp_transformer_layer_cls_to_wrap=None,
x1000c1s1b0n0: full_determinism=False,
x1000c1s1b0n0: gradient_accumulation_steps=1,
x1000c1s1b0n0: gradient_checkpointing=False,
x1000c1s1b0n0: greater_is_better=None,
x1000c1s1b0n0: group_by_length=False,
x1000c1s1b0n0: half_precision_backend=auto,
x1000c1s1b0n0: hub_model_id=None,
x1000c1s1b0n0: hub_private_repo=False,
x1000c1s1b0n0: hub_strategy=every_save,
x1000c1s1b0n0: hub_token=<HUB_TOKEN>,
x1000c1s1b0n0: ignore_data_skip=False,
x1000c1s1b0n0: include_inputs_for_metrics=False,
x1000c1s1b0n0: jit_mode_eval=False,
x1000c1s1b0n0: label_names=None,
x1000c1s1b0n0: label_smoothing_factor=0.0,
x1000c1s1b0n0: learning_rate=5e-05,
x1000c1s1b0n0: length_column_name=length,
x1000c1s1b0n0: load_best_model_at_end=False,
x1000c1s1b0n0: local_rank=0,
x1000c1s1b0n0: log_level=passive,
x1000c1s1b0n0: log_level_replica=warning,
x1000c1s1b0n0: log_on_each_node=True,
x1000c1s1b0n0: logging_dir=/home/users/industry/dso/lannliat/scratch/finetuned_models/debug/runs/Jul16_11-23-46_x1000c1s1b0n0,
x1000c1s1b0n0: logging_first_step=False,
x1000c1s1b0n0: logging_nan_inf_filter=True,
x1000c1s1b0n0: logging_steps=500,
x1000c1s1b0n0: logging_strategy=steps,
x1000c1s1b0n0: lr_scheduler_type=linear,
x1000c1s1b0n0: max_grad_norm=1.0,
x1000c1s1b0n0: max_steps=-1,
x1000c1s1b0n0: metric_for_best_model=None,
x1000c1s1b0n0: mp_parameters=,
x1000c1s1b0n0: no_cuda=False,
x1000c1s1b0n0: num_train_epochs=3.0,
x1000c1s1b0n0: optim=adamw_hf,
x1000c1s1b0n0: optim_args=None,
x1000c1s1b0n0: output_dir=/home/users/industry/dso/lannliat/scratch/finetuned_models/debug,
x1000c1s1b0n0: overwrite_output_dir=False,
x1000c1s1b0n0: past_index=-1,
x1000c1s1b0n0: per_device_eval_batch_size=8,
x1000c1s1b0n0: per_device_train_batch_size=8,
x1000c1s1b0n0: prediction_loss_only=False,
x1000c1s1b0n0: push_to_hub=False,
x1000c1s1b0n0: push_to_hub_model_id=None,
x1000c1s1b0n0: push_to_hub_organization=None,
x1000c1s1b0n0: push_to_hub_token=<PUSH_TO_HUB_TOKEN>,
x1000c1s1b0n0: ray_scope=last,
x1000c1s1b0n0: remove_unused_columns=True,
x1000c1s1b0n0: report_to=['wandb'],
x1000c1s1b0n0: resume_from_checkpoint=None,
x1000c1s1b0n0: run_name=/home/users/industry/dso/lannliat/scratch/finetuned_models/debug,
x1000c1s1b0n0: save_on_each_node=False,
x1000c1s1b0n0: save_safetensors=False,
x1000c1s1b0n0: save_steps=500,
x1000c1s1b0n0: save_strategy=steps,
x1000c1s1b0n0: save_total_limit=None,
x1000c1s1b0n0: seed=42,
x1000c1s1b0n0: sharded_ddp=[],
x1000c1s1b0n0: skip_memory_metrics=True,
x1000c1s1b0n0: tf32=None,
x1000c1s1b0n0: torch_compile=False,
x1000c1s1b0n0: torch_compile_backend=None,
x1000c1s1b0n0: torch_compile_mode=None,
x1000c1s1b0n0: torchdynamo=None,
x1000c1s1b0n0: tpu_metrics_debug=False,
x1000c1s1b0n0: tpu_num_cores=None,
x1000c1s1b0n0: use_ipex=False,
x1000c1s1b0n0: use_legacy_prediction_loop=False,
x1000c1s1b0n0: use_mps_device=False,
x1000c1s1b0n0: warmup_ratio=0.0,
x1000c1s1b0n0: warmup_steps=0,
x1000c1s1b0n0: weight_decay=0.0,
x1000c1s1b0n0: xpu_backend=None,
x1000c1s1b0n0: )
x1000c1s1b0n0: 07/16/2023 11:23:47 - WARNING - __main__ - Process rank: 1, device: cuda:1, n_gpu: 1distributed training: True, 16-bits training: False
x1000c1s1b0n0: 07/16/2023 11:23:49 - INFO - datasets.info - Loading Dataset Infos from /home/users/industry/dso/lannliat/.cache/huggingface/modules/datasets_modules/datasets/wikitext/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126
x1000c1s1b0n0: 07/16/2023 11:23:49 - INFO - datasets.builder - Overwrite dataset info from restored data version if exists.
x1000c1s1b0n0: 07/16/2023 11:23:49 - INFO - datasets.info - Loading Dataset info from /home/users/industry/dso/lannliat/.cache/huggingface/datasets/wikitext/wikitext-2-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126
x1000c1s1b0n0: 07/16/2023 11:23:49 - WARNING - datasets.builder - Found cached dataset wikitext (/home/users/industry/dso/lannliat/.cache/huggingface/datasets/wikitext/wikitext-2-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126)
x1000c1s1b0n0: 07/16/2023 11:23:49 - INFO - datasets.info - Loading Dataset info from /home/users/industry/dso/lannliat/.cache/huggingface/datasets/wikitext/wikitext-2-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126
100%|██████████| 3/3 [00:00<00:00, 272.09it/s]
x1000c0s6b0n0: 07/16/2023 11:23:50 - INFO - datasets.info - Loading Dataset Infos from /home/users/industry/dso/lannliat/.cache/huggingface/modules/datasets_modules/datasets/wikitext/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126
x1000c0s6b0n0: 07/16/2023 11:23:50 - INFO - datasets.builder - Overwrite dataset info from restored data version if exists.
x1000c0s6b0n0: 07/16/2023 11:23:50 - INFO - datasets.info - Loading Dataset info from /home/users/industry/dso/lannliat/.cache/huggingface/datasets/wikitext/wikitext-2-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126
x1000c0s6b0n0: 07/16/2023 11:23:50 - WARNING - datasets.builder - Found cached dataset wikitext (/home/users/industry/dso/lannliat/.cache/huggingface/datasets/wikitext/wikitext-2-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126)
100%|██████████| 3/3 [00:00<00:00, 194.61it/s]
x1000c0s6b0n0: 07/16/2023 11:23:50 - WARNING - datasets.builder - Found cached dataset wikitext (/home/users/industry/dso/lannliat/.cache/huggingface/datasets/wikitext/wikitext-2-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126)
x1000c0s6b0n0: 07/16/2023 11:23:50 - INFO - datasets.info - Loading Dataset info from /home/users/industry/dso/lannliat/.cache/huggingface/datasets/wikitext/wikitext-2-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126
100%|██████████| 3/3 [00:00<00:00, 811.12it/s]
x1000c1s1b0n0: [INFO|configuration_utils.py:712] 2023-07-16 11:23:50,641 >> loading configuration file config.json from cache at /home/users/industry/dso/lannliat/.cache/huggingface/hub/models--gpt2/snapshots/11c5a3d5811f50298f278a704980280950aedb10/config.json
x1000c1s1b0n0: [INFO|configuration_utils.py:768] 2023-07-16 11:23:50,641 >> Model config GPT2Config {
x1000c1s1b0n0:   "_name_or_path": "gpt2",
x1000c1s1b0n0:   "activation_function": "gelu_new",
x1000c1s1b0n0:   "architectures": [
x1000c1s1b0n0:     "GPT2LMHeadModel"
x1000c1s1b0n0:   ],
x1000c1s1b0n0:   "attn_pdrop": 0.1,
x1000c1s1b0n0:   "bos_token_id": 50256,
x1000c1s1b0n0:   "embd_pdrop": 0.1,
x1000c1s1b0n0:   "eos_token_id": 50256,
x1000c1s1b0n0:   "initializer_range": 0.02,
x1000c1s1b0n0:   "layer_norm_epsilon": 1e-05,
x1000c1s1b0n0:   "model_type": "gpt2",
x1000c1s1b0n0:   "n_ctx": 1024,
x1000c1s1b0n0:   "n_embd": 768,
x1000c1s1b0n0:   "n_head": 12,
x1000c1s1b0n0:   "n_inner": null,
x1000c1s1b0n0:   "n_layer": 12,
x1000c1s1b0n0:   "n_positions": 1024,
x1000c1s1b0n0:   "reorder_and_upcast_attn": false,
x1000c1s1b0n0:   "resid_pdrop": 0.1,
x1000c1s1b0n0:   "scale_attn_by_inverse_layer_idx": false,
x1000c1s1b0n0:   "scale_attn_weights": true,
x1000c1s1b0n0:   "summary_activation": null,
x1000c1s1b0n0:   "summary_first_dropout": 0.1,
x1000c1s1b0n0:   "summary_proj_to_labels": true,
x1000c1s1b0n0:   "summary_type": "cls_index",
x1000c1s1b0n0:   "summary_use_proj": true,
x1000c1s1b0n0:   "task_specific_params": {
x1000c1s1b0n0:     "text-generation": {
x1000c1s1b0n0:       "do_sample": true,
x1000c1s1b0n0:       "max_length": 50
x1000c1s1b0n0:     }
x1000c1s1b0n0:   },
x1000c1s1b0n0:   "transformers_version": "4.31.0.dev0",
x1000c1s1b0n0:   "use_cache": true,
x1000c1s1b0n0:   "vocab_size": 50257
x1000c1s1b0n0: }
x1000c1s1b0n0: 
x1000c0s6b0n0: [INFO|configuration_utils.py:712] 2023-07-16 11:23:50,647 >> loading configuration file config.json from cache at /home/users/industry/dso/lannliat/.cache/huggingface/hub/models--gpt2/snapshots/11c5a3d5811f50298f278a704980280950aedb10/config.json
x1000c0s6b0n0: [INFO|configuration_utils.py:768] 2023-07-16 11:23:50,648 >> Model config GPT2Config {
x1000c0s6b0n0:   "_name_or_path": "gpt2",
x1000c0s6b0n0:   "activation_function": "gelu_new",
x1000c0s6b0n0:   "architectures": [
x1000c0s6b0n0:     "GPT2LMHeadModel"
x1000c0s6b0n0:   ],
x1000c0s6b0n0:   "attn_pdrop": 0.1,
x1000c0s6b0n0:   "bos_token_id": 50256,
x1000c0s6b0n0:   "embd_pdrop": 0.1,
x1000c0s6b0n0:   "eos_token_id": 50256,
x1000c0s6b0n0:   "initializer_range": 0.02,
x1000c0s6b0n0:   "layer_norm_epsilon": 1e-05,
x1000c0s6b0n0:   "model_type": "gpt2",
x1000c0s6b0n0:   "n_ctx": 1024,
x1000c0s6b0n0:   "n_embd": 768,
x1000c0s6b0n0:   "n_head": 12,
x1000c0s6b0n0:   "n_inner": null,
x1000c0s6b0n0:   "n_layer": 12,
x1000c0s6b0n0:   "n_positions": 1024,
x1000c0s6b0n0:   "reorder_and_upcast_attn": false,
x1000c0s6b0n0:   "resid_pdrop": 0.1,
x1000c0s6b0n0:   "scale_attn_by_inverse_layer_idx": false,
x1000c0s6b0n0:   "scale_attn_weights": true,
x1000c0s6b0n0:   "summary_activation": null,
x1000c0s6b0n0:   "summary_first_dropout": 0.1,
x1000c0s6b0n0:   "summary_proj_to_labels": true,
x1000c0s6b0n0:   "summary_type": "cls_index",
x1000c0s6b0n0:   "summary_use_proj": true,
x1000c0s6b0n0:   "task_specific_params": {
x1000c0s6b0n0:     "text-generation": {
x1000c0s6b0n0:       "do_sample": true,
x1000c0s6b0n0:       "max_length": 50
x1000c0s6b0n0:     }
x1000c0s6b0n0:   },
x1000c0s6b0n0:   "transformers_version": "4.31.0.dev0",
x1000c0s6b0n0:   "use_cache": true,
x1000c0s6b0n0:   "vocab_size": 50257
x1000c0s6b0n0: }
x1000c0s6b0n0: 
x1000c1s1b0n0: 07/16/2023 11:23:50 - WARNING - datasets.builder - Found cached dataset wikitext (/home/users/industry/dso/lannliat/.cache/huggingface/datasets/wikitext/wikitext-2-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126)
100%|██████████| 3/3 [00:00<00:00, 1171.27it/s]
x1000c0s6b0n0: [INFO|tokenization_auto.py:512] 2023-07-16 11:23:50,890 >> Could not locate the tokenizer configuration file, will try to use the model config instead.
x1000c0s6b0n0: [INFO|configuration_utils.py:712] 2023-07-16 11:23:51,134 >> loading configuration file config.json from cache at /home/users/industry/dso/lannliat/.cache/huggingface/hub/models--gpt2/snapshots/11c5a3d5811f50298f278a704980280950aedb10/config.json
x1000c0s6b0n0: [INFO|configuration_utils.py:768] 2023-07-16 11:23:51,135 >> Model config GPT2Config {
x1000c0s6b0n0:   "_name_or_path": "gpt2",
x1000c0s6b0n0:   "activation_function": "gelu_new",
x1000c0s6b0n0:   "architectures": [
x1000c0s6b0n0:     "GPT2LMHeadModel"
x1000c0s6b0n0:   ],
x1000c0s6b0n0:   "attn_pdrop": 0.1,
x1000c0s6b0n0:   "bos_token_id": 50256,
x1000c0s6b0n0:   "embd_pdrop": 0.1,
x1000c0s6b0n0:   "eos_token_id": 50256,
x1000c0s6b0n0:   "initializer_range": 0.02,
x1000c0s6b0n0:   "layer_norm_epsilon": 1e-05,
x1000c0s6b0n0:   "model_type": "gpt2",
x1000c0s6b0n0:   "n_ctx": 1024,
x1000c0s6b0n0:   "n_embd": 768,
x1000c0s6b0n0:   "n_head": 12,
x1000c0s6b0n0:   "n_inner": null,
x1000c0s6b0n0:   "n_layer": 12,
x1000c0s6b0n0:   "n_positions": 1024,
x1000c0s6b0n0:   "reorder_and_upcast_attn": false,
x1000c0s6b0n0:   "resid_pdrop": 0.1,
x1000c0s6b0n0:   "scale_attn_by_inverse_layer_idx": false,
x1000c0s6b0n0:   "scale_attn_weights": true,
x1000c0s6b0n0:   "summary_activation": null,
x1000c0s6b0n0:   "summary_first_dropout": 0.1,
x1000c0s6b0n0:   "summary_proj_to_labels": true,
x1000c0s6b0n0:   "summary_type": "cls_index",
x1000c0s6b0n0:   "summary_use_proj": true,
x1000c0s6b0n0:   "task_specific_params": {
x1000c0s6b0n0:     "text-generation": {
x1000c0s6b0n0:       "do_sample": true,
x1000c0s6b0n0:       "max_length": 50
x1000c0s6b0n0:     }
x1000c0s6b0n0:   },
x1000c0s6b0n0:   "transformers_version": "4.31.0.dev0",
x1000c0s6b0n0:   "use_cache": true,
x1000c0s6b0n0:   "vocab_size": 50257
x1000c0s6b0n0: }
x1000c0s6b0n0: 
x1000c1s1b0n0: [INFO|tokenization_auto.py:512] 2023-07-16 11:23:51,338 >> Could not locate the tokenizer configuration file, will try to use the model config instead.
x1000c1s1b0n0: [INFO|configuration_utils.py:712] 2023-07-16 11:23:51,581 >> loading configuration file config.json from cache at /home/users/industry/dso/lannliat/.cache/huggingface/hub/models--gpt2/snapshots/11c5a3d5811f50298f278a704980280950aedb10/config.json
x1000c1s1b0n0: [INFO|configuration_utils.py:768] 2023-07-16 11:23:51,582 >> Model config GPT2Config {
x1000c1s1b0n0:   "_name_or_path": "gpt2",
x1000c1s1b0n0:   "activation_function": "gelu_new",
x1000c1s1b0n0:   "architectures": [
x1000c1s1b0n0:     "GPT2LMHeadModel"
x1000c1s1b0n0:   ],
x1000c1s1b0n0:   "attn_pdrop": 0.1,
x1000c1s1b0n0:   "bos_token_id": 50256,
x1000c1s1b0n0:   "embd_pdrop": 0.1,
x1000c1s1b0n0:   "eos_token_id": 50256,
x1000c1s1b0n0:   "initializer_range": 0.02,
x1000c1s1b0n0:   "layer_norm_epsilon": 1e-05,
x1000c1s1b0n0:   "model_type": "gpt2",
x1000c1s1b0n0:   "n_ctx": 1024,
x1000c1s1b0n0:   "n_embd": 768,
x1000c1s1b0n0:   "n_head": 12,
x1000c1s1b0n0:   "n_inner": null,
x1000c1s1b0n0:   "n_layer": 12,
x1000c1s1b0n0:   "n_positions": 1024,
x1000c1s1b0n0:   "reorder_and_upcast_attn": false,
x1000c1s1b0n0:   "resid_pdrop": 0.1,
x1000c1s1b0n0:   "scale_attn_by_inverse_layer_idx": false,
x1000c1s1b0n0:   "scale_attn_weights": true,
x1000c1s1b0n0:   "summary_activation": null,
x1000c1s1b0n0:   "summary_first_dropout": 0.1,
x1000c1s1b0n0:   "summary_proj_to_labels": true,
x1000c1s1b0n0:   "summary_type": "cls_index",
x1000c1s1b0n0:   "summary_use_proj": true,
x1000c1s1b0n0:   "task_specific_params": {
x1000c1s1b0n0:     "text-generation": {
x1000c1s1b0n0:       "do_sample": true,
x1000c1s1b0n0:       "max_length": 50
x1000c1s1b0n0:     }
x1000c1s1b0n0:   },
x1000c1s1b0n0:   "transformers_version": "4.31.0.dev0",
x1000c1s1b0n0:   "use_cache": true,
x1000c1s1b0n0:   "vocab_size": 50257
x1000c1s1b0n0: }
x1000c1s1b0n0: 
x1000c0s6b0n0: [INFO|tokenization_utils_base.py:1843] 2023-07-16 11:23:52,137 >> loading file vocab.json from cache at /home/users/industry/dso/lannliat/.cache/huggingface/hub/models--gpt2/snapshots/11c5a3d5811f50298f278a704980280950aedb10/vocab.json
x1000c0s6b0n0: [INFO|tokenization_utils_base.py:1843] 2023-07-16 11:23:52,137 >> loading file merges.txt from cache at /home/users/industry/dso/lannliat/.cache/huggingface/hub/models--gpt2/snapshots/11c5a3d5811f50298f278a704980280950aedb10/merges.txt
x1000c0s6b0n0: [INFO|tokenization_utils_base.py:1843] 2023-07-16 11:23:52,137 >> loading file tokenizer.json from cache at /home/users/industry/dso/lannliat/.cache/huggingface/hub/models--gpt2/snapshots/11c5a3d5811f50298f278a704980280950aedb10/tokenizer.json
x1000c0s6b0n0: [INFO|tokenization_utils_base.py:1843] 2023-07-16 11:23:52,137 >> loading file added_tokens.json from cache at None
x1000c0s6b0n0: [INFO|tokenization_utils_base.py:1843] 2023-07-16 11:23:52,137 >> loading file special_tokens_map.json from cache at None
x1000c0s6b0n0: [INFO|tokenization_utils_base.py:1843] 2023-07-16 11:23:52,137 >> loading file tokenizer_config.json from cache at None
x1000c0s6b0n0: [INFO|configuration_utils.py:712] 2023-07-16 11:23:52,139 >> loading configuration file config.json from cache at /home/users/industry/dso/lannliat/.cache/huggingface/hub/models--gpt2/snapshots/11c5a3d5811f50298f278a704980280950aedb10/config.json
x1000c0s6b0n0: [INFO|configuration_utils.py:768] 2023-07-16 11:23:52,140 >> Model config GPT2Config {
x1000c0s6b0n0:   "_name_or_path": "gpt2",
x1000c0s6b0n0:   "activation_function": "gelu_new",
x1000c0s6b0n0:   "architectures": [
x1000c0s6b0n0:     "GPT2LMHeadModel"
x1000c0s6b0n0:   ],
x1000c0s6b0n0:   "attn_pdrop": 0.1,
x1000c0s6b0n0:   "bos_token_id": 50256,
x1000c0s6b0n0:   "embd_pdrop": 0.1,
x1000c0s6b0n0:   "eos_token_id": 50256,
x1000c0s6b0n0:   "initializer_range": 0.02,
x1000c0s6b0n0:   "layer_norm_epsilon": 1e-05,
x1000c0s6b0n0:   "model_type": "gpt2",
x1000c0s6b0n0:   "n_ctx": 1024,
x1000c0s6b0n0:   "n_embd": 768,
x1000c0s6b0n0:   "n_head": 12,
x1000c0s6b0n0:   "n_inner": null,
x1000c0s6b0n0:   "n_layer": 12,
x1000c0s6b0n0:   "n_positions": 1024,
x1000c0s6b0n0:   "reorder_and_upcast_attn": false,
x1000c0s6b0n0:   "resid_pdrop": 0.1,
x1000c0s6b0n0:   "scale_attn_by_inverse_layer_idx": false,
x1000c0s6b0n0:   "scale_attn_weights": true,
x1000c0s6b0n0:   "summary_activation": null,
x1000c0s6b0n0:   "summary_first_dropout": 0.1,
x1000c0s6b0n0:   "summary_proj_to_labels": true,
x1000c0s6b0n0:   "summary_type": "cls_index",
x1000c0s6b0n0:   "summary_use_proj": true,
x1000c0s6b0n0:   "task_specific_params": {
x1000c0s6b0n0:     "text-generation": {
x1000c0s6b0n0:       "do_sample": true,
x1000c0s6b0n0:       "max_length": 50
x1000c0s6b0n0:     }
x1000c0s6b0n0:   },
x1000c0s6b0n0:   "transformers_version": "4.31.0.dev0",
x1000c0s6b0n0:   "use_cache": true,
x1000c0s6b0n0:   "vocab_size": 50257
x1000c0s6b0n0: }
x1000c0s6b0n0: 
x1000c0s6b0n0: [INFO|modeling_utils.py:2603] 2023-07-16 11:23:52,218 >> loading weights file model.safetensors from cache at /home/users/industry/dso/lannliat/.cache/huggingface/hub/models--gpt2/snapshots/11c5a3d5811f50298f278a704980280950aedb10/model.safetensors
x1000c1s1b0n0: [INFO|tokenization_utils_base.py:1843] 2023-07-16 11:23:52,571 >> loading file vocab.json from cache at /home/users/industry/dso/lannliat/.cache/huggingface/hub/models--gpt2/snapshots/11c5a3d5811f50298f278a704980280950aedb10/vocab.json
x1000c1s1b0n0: [INFO|tokenization_utils_base.py:1843] 2023-07-16 11:23:52,571 >> loading file merges.txt from cache at /home/users/industry/dso/lannliat/.cache/huggingface/hub/models--gpt2/snapshots/11c5a3d5811f50298f278a704980280950aedb10/merges.txt
x1000c1s1b0n0: [INFO|tokenization_utils_base.py:1843] 2023-07-16 11:23:52,571 >> loading file tokenizer.json from cache at /home/users/industry/dso/lannliat/.cache/huggingface/hub/models--gpt2/snapshots/11c5a3d5811f50298f278a704980280950aedb10/tokenizer.json
x1000c1s1b0n0: [INFO|tokenization_utils_base.py:1843] 2023-07-16 11:23:52,571 >> loading file added_tokens.json from cache at None
x1000c1s1b0n0: [INFO|tokenization_utils_base.py:1843] 2023-07-16 11:23:52,571 >> loading file special_tokens_map.json from cache at None
x1000c1s1b0n0: [INFO|tokenization_utils_base.py:1843] 2023-07-16 11:23:52,571 >> loading file tokenizer_config.json from cache at None
x1000c1s1b0n0: [INFO|configuration_utils.py:712] 2023-07-16 11:23:52,573 >> loading configuration file config.json from cache at /home/users/industry/dso/lannliat/.cache/huggingface/hub/models--gpt2/snapshots/11c5a3d5811f50298f278a704980280950aedb10/config.json
x1000c1s1b0n0: [INFO|configuration_utils.py:768] 2023-07-16 11:23:52,574 >> Model config GPT2Config {
x1000c1s1b0n0:   "_name_or_path": "gpt2",
x1000c1s1b0n0:   "activation_function": "gelu_new",
x1000c1s1b0n0:   "architectures": [
x1000c1s1b0n0:     "GPT2LMHeadModel"
x1000c1s1b0n0:   ],
x1000c1s1b0n0:   "attn_pdrop": 0.1,
x1000c1s1b0n0:   "bos_token_id": 50256,
x1000c1s1b0n0:   "embd_pdrop": 0.1,
x1000c1s1b0n0:   "eos_token_id": 50256,
x1000c1s1b0n0:   "initializer_range": 0.02,
x1000c1s1b0n0:   "layer_norm_epsilon": 1e-05,
x1000c1s1b0n0:   "model_type": "gpt2",
x1000c1s1b0n0:   "n_ctx": 1024,
x1000c1s1b0n0:   "n_embd": 768,
x1000c1s1b0n0:   "n_head": 12,
x1000c1s1b0n0:   "n_inner": null,
x1000c1s1b0n0:   "n_layer": 12,
x1000c1s1b0n0:   "n_positions": 1024,
x1000c1s1b0n0:   "reorder_and_upcast_attn": false,
x1000c1s1b0n0:   "resid_pdrop": 0.1,
x1000c1s1b0n0:   "scale_attn_by_inverse_layer_idx": false,
x1000c1s1b0n0:   "scale_attn_weights": true,
x1000c1s1b0n0:   "summary_activation": null,
x1000c1s1b0n0:   "summary_first_dropout": 0.1,
x1000c1s1b0n0:   "summary_proj_to_labels": true,
x1000c1s1b0n0:   "summary_type": "cls_index",
x1000c1s1b0n0:   "summary_use_proj": true,
x1000c1s1b0n0:   "task_specific_params": {
x1000c1s1b0n0:     "text-generation": {
x1000c1s1b0n0:       "do_sample": true,
x1000c1s1b0n0:       "max_length": 50
x1000c1s1b0n0:     }
x1000c1s1b0n0:   },
x1000c1s1b0n0:   "transformers_version": "4.31.0.dev0",
x1000c1s1b0n0:   "use_cache": true,
x1000c1s1b0n0:   "vocab_size": 50257
x1000c1s1b0n0: }
x1000c1s1b0n0: 
x1000c1s1b0n0: [INFO|modeling_utils.py:2603] 2023-07-16 11:23:52,649 >> loading weights file model.safetensors from cache at /home/users/industry/dso/lannliat/.cache/huggingface/hub/models--gpt2/snapshots/11c5a3d5811f50298f278a704980280950aedb10/model.safetensors
x1000c1s1b0n0: [INFO|modeling_utils.py:2694] 2023-07-16 11:23:52,754 >> Detected DeepSpeed ZeRO-3: activating zero.init() for this model
x1000c0s6b0n0: [INFO|modeling_utils.py:2694] 2023-07-16 11:23:52,754 >> Detected DeepSpeed ZeRO-3: activating zero.init() for this model
x1000c1s1b0n0: [INFO|configuration_utils.py:599] 2023-07-16 11:23:52,757 >> Generate config GenerationConfig {
x1000c1s1b0n0:   "_from_model_config": true,
x1000c1s1b0n0:   "bos_token_id": 50256,
x1000c1s1b0n0:   "eos_token_id": 50256,
x1000c1s1b0n0:   "transformers_version": "4.31.0.dev0"
x1000c1s1b0n0: }
x1000c1s1b0n0: 
x1000c0s6b0n0: [INFO|configuration_utils.py:599] 2023-07-16 11:23:52,759 >> Generate config GenerationConfig {
x1000c0s6b0n0:   "_from_model_config": true,
x1000c0s6b0n0:   "bos_token_id": 50256,
x1000c0s6b0n0:   "eos_token_id": 50256,
x1000c0s6b0n0:   "transformers_version": "4.31.0.dev0"
x1000c0s6b0n0: }
x1000c0s6b0n0: 
x1000c0s6b0n0: [2023-07-16 11:23:58,743] [INFO] [partition_parameters.py:453:__exit__] finished initializing model with 0.16B parameters
x1000c1s1b0n0: [INFO|modeling_utils.py:3329] 2023-07-16 11:23:59,764 >> All model checkpoint weights were used when initializing GPT2LMHeadModel.
x1000c1s1b0n0: 
x1000c0s6b0n0: [INFO|modeling_utils.py:3329] 2023-07-16 11:23:59,764 >> All model checkpoint weights were used when initializing GPT2LMHeadModel.
x1000c1s1b0n0: [INFO|modeling_utils.py:3337] 2023-07-16 11:23:59,764 >> All the weights of GPT2LMHeadModel were initialized from the model checkpoint at gpt2.
x1000c0s6b0n0: 
x1000c1s1b0n0: If your task is similar to the task the model of the checkpoint was trained on, you can already use GPT2LMHeadModel for predictions without further training.
x1000c0s6b0n0: [INFO|modeling_utils.py:3337] 2023-07-16 11:23:59,764 >> All the weights of GPT2LMHeadModel were initialized from the model checkpoint at gpt2.
x1000c0s6b0n0: If your task is similar to the task the model of the checkpoint was trained on, you can already use GPT2LMHeadModel for predictions without further training.
x1000c0s6b0n0: [INFO|configuration_utils.py:561] 2023-07-16 11:24:00,009 >> loading configuration file generation_config.json from cache at /home/users/industry/dso/lannliat/.cache/huggingface/hub/models--gpt2/snapshots/11c5a3d5811f50298f278a704980280950aedb10/generation_config.json
x1000c0s6b0n0: [INFO|configuration_utils.py:599] 2023-07-16 11:24:00,009 >> Generate config GenerationConfig {
x1000c0s6b0n0:   "_from_model_config": true,
x1000c0s6b0n0:   "bos_token_id": 50256,
x1000c0s6b0n0:   "eos_token_id": 50256,
x1000c0s6b0n0:   "transformers_version": "4.31.0.dev0"
x1000c0s6b0n0: }
x1000c0s6b0n0: 
x1000c1s1b0n0: [INFO|configuration_utils.py:561] 2023-07-16 11:24:00,009 >> loading configuration file generation_config.json from cache at /home/users/industry/dso/lannliat/.cache/huggingface/hub/models--gpt2/snapshots/11c5a3d5811f50298f278a704980280950aedb10/generation_config.json
x1000c1s1b0n0: [INFO|configuration_utils.py:599] 2023-07-16 11:24:00,010 >> Generate config GenerationConfig {
x1000c1s1b0n0:   "_from_model_config": true,
x1000c1s1b0n0:   "bos_token_id": 50256,
x1000c1s1b0n0:   "eos_token_id": 50256,
x1000c1s1b0n0:   "transformers_version": "4.31.0.dev0"
x1000c1s1b0n0: }
x1000c1s1b0n0: 
x1000c1s1b0n0: 07/16/2023 11:24:00 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/users/industry/dso/lannliat/.cache/huggingface/datasets/wikitext/wikitext-2-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126/cache-44ec8a7bce9ef049.arrow
x1000c1s1b0n0: 07/16/2023 11:24:00 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/users/industry/dso/lannliat/.cache/huggingface/datasets/wikitext/wikitext-2-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126/cache-721cda7e77511ffc.arrow
x1000c0s6b0n0: 07/16/2023 11:24:00 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/users/industry/dso/lannliat/.cache/huggingface/datasets/wikitext/wikitext-2-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126/cache-44ec8a7bce9ef049.arrow
x1000c0s6b0n0: 07/16/2023 11:24:00 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/users/industry/dso/lannliat/.cache/huggingface/datasets/wikitext/wikitext-2-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126/cache-721cda7e77511ffc.arrow
x1000c1s1b0n0: 07/16/2023 11:24:00 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/users/industry/dso/lannliat/.cache/huggingface/datasets/wikitext/wikitext-2-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126/cache-49f826bee8b8a100.arrow
x1000c0s6b0n0: 07/16/2023 11:24:00 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/users/industry/dso/lannliat/.cache/huggingface/datasets/wikitext/wikitext-2-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126/cache-49f826bee8b8a100.arrow
x1000c1s1b0n0: 07/16/2023 11:24:00 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/users/industry/dso/lannliat/.cache/huggingface/datasets/wikitext/wikitext-2-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126/cache-c87119ecc69384c8.arrow
x1000c0s6b0n0: 07/16/2023 11:24:00 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/users/industry/dso/lannliat/.cache/huggingface/datasets/wikitext/wikitext-2-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126/cache-c87119ecc69384c8.arrow
x1000c1s1b0n0: 07/16/2023 11:24:00 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/users/industry/dso/lannliat/.cache/huggingface/datasets/wikitext/wikitext-2-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126/cache-44ec8a7bce9ef049.arrow
x1000c0s6b0n0: 07/16/2023 11:24:00 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/users/industry/dso/lannliat/.cache/huggingface/datasets/wikitext/wikitext-2-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126/cache-44ec8a7bce9ef049.arrow
x1000c1s1b0n0: 07/16/2023 11:24:00 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/users/industry/dso/lannliat/.cache/huggingface/datasets/wikitext/wikitext-2-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126/cache-721cda7e77511ffc.arrow
x1000c1s1b0n0: 07/16/2023 11:24:00 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/users/industry/dso/lannliat/.cache/huggingface/datasets/wikitext/wikitext-2-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126/cache-49f826bee8b8a100.arrow
x1000c0s6b0n0: 07/16/2023 11:24:00 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/users/industry/dso/lannliat/.cache/huggingface/datasets/wikitext/wikitext-2-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126/cache-721cda7e77511ffc.arrow
x1000c0s6b0n0: 07/16/2023 11:24:00 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/users/industry/dso/lannliat/.cache/huggingface/datasets/wikitext/wikitext-2-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126/cache-49f826bee8b8a100.arrow
x1000c1s1b0n0: 07/16/2023 11:24:00 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/users/industry/dso/lannliat/.cache/huggingface/datasets/wikitext/wikitext-2-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126/cache-acd38b65189dc44f.arrow
x1000c0s6b0n0: 07/16/2023 11:24:00 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/users/industry/dso/lannliat/.cache/huggingface/datasets/wikitext/wikitext-2-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126/cache-acd38b65189dc44f.arrow
x1000c1s1b0n0: 07/16/2023 11:24:00 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/users/industry/dso/lannliat/.cache/huggingface/datasets/wikitext/wikitext-2-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126/cache-c1cf316f13c4acf5.arrow
x1000c0s6b0n0: 07/16/2023 11:24:00 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/users/industry/dso/lannliat/.cache/huggingface/datasets/wikitext/wikitext-2-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126/cache-c1cf316f13c4acf5.arrow
x1000c1s1b0n0: 07/16/2023 11:24:00 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/users/industry/dso/lannliat/.cache/huggingface/datasets/wikitext/wikitext-2-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126/cache-c87119ecc69384c8.arrow
x1000c0s6b0n0: 07/16/2023 11:24:00 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/users/industry/dso/lannliat/.cache/huggingface/datasets/wikitext/wikitext-2-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126/cache-c87119ecc69384c8.arrow
x1000c1s1b0n0: 07/16/2023 11:24:00 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/users/industry/dso/lannliat/.cache/huggingface/datasets/wikitext/wikitext-2-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126/cache-acd38b65189dc44f.arrow
x1000c0s6b0n0: 07/16/2023 11:24:00 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/users/industry/dso/lannliat/.cache/huggingface/datasets/wikitext/wikitext-2-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126/cache-acd38b65189dc44f.arrow
x1000c1s1b0n0: 07/16/2023 11:24:00 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/users/industry/dso/lannliat/.cache/huggingface/datasets/wikitext/wikitext-2-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126/cache-c1cf316f13c4acf5.arrow
x1000c0s6b0n0: 07/16/2023 11:24:00 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/users/industry/dso/lannliat/.cache/huggingface/datasets/wikitext/wikitext-2-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126/cache-c1cf316f13c4acf5.arrow
x1000c0s6b0n0: [2023-07-16 11:24:01,734] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed info: version=0.9.5, git-hash=unknown, git-branch=unknown
x1000c0s6b0n0: [2023-07-16 11:24:02,277] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False
x1000c0s6b0n0:  [WARNING]  cpu_adam cuda is missing or is incompatible with installed torch, only cpu ops can be compiled!
x1000c0s6b0n0: Using /home/users/industry/dso/lannliat/.cache/torch_extensions/py310_cu116 as PyTorch extensions root...
x1000c1s1b0n0:  [WARNING]  cpu_adam cuda is missing or is incompatible with installed torch, only cpu ops can be compiled!
x1000c1s1b0n0: Using /home/users/industry/dso/lannliat/.cache/torch_extensions/py310_cu116 as PyTorch extensions root...
x1000c1s1b0n0:  [WARNING]  cpu_adam cuda is missing or is incompatible with installed torch, only cpu ops can be compiled!
x1000c1s1b0n0: Using /home/users/industry/dso/lannliat/.cache/torch_extensions/py310_cu116 as PyTorch extensions root...
x1000c1s1b0n0: Traceback (most recent call last):
x1000c1s1b0n0:   File "/home/users/industry/dso/lannliat/transformers/examples/pytorch/language-modeling/run_clm.py", line 634, in <module>
x1000c1s1b0n0:     main()
x1000c1s1b0n0:   File "/home/users/industry/dso/lannliat/transformers/examples/pytorch/language-modeling/run_clm.py", line 582, in main
x1000c1s1b0n0:     train_result = trainer.train(resume_from_checkpoint=checkpoint)
x1000c1s1b0n0:   File "/home/users/industry/dso/lannliat/transformers/src/transformers/trainer.py", line 1539, in train
x1000c1s1b0n0:     return inner_training_loop(
x1000c1s1b0n0:   File "/home/users/industry/dso/lannliat/transformers/src/transformers/trainer.py", line 1659, in _inner_training_loop
x1000c1s1b0n0:     model, self.optimizer, self.lr_scheduler = self.accelerator.prepare(
x1000c1s1b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/accelerate/accelerator.py", line 1198, in prepare
x1000c1s1b0n0:     result = self._prepare_deepspeed(*args)
x1000c1s1b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/accelerate/accelerator.py", line 1537, in _prepare_deepspeed
x1000c1s1b0n0:     engine, optimizer, _, lr_scheduler = deepspeed.initialize(**kwargs)
x1000c1s1b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/deepspeed/__init__.py", line 165, in initialize
x1000c1s1b0n0:     engine = DeepSpeedEngine(args=args,
x1000c1s1b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 309, in __init__
x1000c1s1b0n0:     self._configure_optimizer(optimizer, model_parameters)
x1000c1s1b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 1173, in _configure_optimizer
x1000c1s1b0n0:     basic_optimizer = self._configure_basic_optimizer(model_parameters)
x1000c1s1b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 1229, in _configure_basic_optimizer
x1000c1s1b0n0:     optimizer = DeepSpeedCPUAdam(model_parameters,
x1000c1s1b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/deepspeed/ops/adam/cpu_adam.py", line 94, in __init__
x1000c1s1b0n0:     self.ds_opt_adam = CPUAdamBuilder().load()
x1000c1s1b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/deepspeed/ops/op_builder/builder.py", line 454, in load
x1000c1s1b0n0:     return self.jit_load(verbose)
x1000c1s1b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/deepspeed/ops/op_builder/builder.py", line 497, in jit_load
x1000c1s1b0n0:     op_module = load(name=self.name,
x1000c1s1b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1284, in load
x1000c1s1b0n0:     return _jit_compile(
x1000c1s1b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1508, in _jit_compile
x1000c1s1b0n0:     _write_ninja_file_and_build_library(
x1000c1s1b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1592, in _write_ninja_file_and_build_library
x1000c1s1b0n0:     verify_ninja_availability()
x1000c1s1b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1648, in verify_ninja_availability
x1000c1s1b0n0:     raise RuntimeError("Ninja is required to load C++ extensions")
x1000c1s1b0n0: RuntimeError: Ninja is required to load C++ extensions
x1000c0s6b0n0: Traceback (most recent call last):
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/transformers/examples/pytorch/language-modeling/run_clm.py", line 634, in <module>
x1000c0s6b0n0:     main()
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/transformers/examples/pytorch/language-modeling/run_clm.py", line 582, in main
x1000c0s6b0n0:     train_result = trainer.train(resume_from_checkpoint=checkpoint)
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/transformers/src/transformers/trainer.py", line 1539, in train
x1000c0s6b0n0:     return inner_training_loop(
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/transformers/src/transformers/trainer.py", line 1659, in _inner_training_loop
x1000c0s6b0n0:     model, self.optimizer, self.lr_scheduler = self.accelerator.prepare(
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/accelerate/accelerator.py", line 1198, in prepare
x1000c0s6b0n0:     result = self._prepare_deepspeed(*args)
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/accelerate/accelerator.py", line 1537, in _prepare_deepspeed
x1000c0s6b0n0:     engine, optimizer, _, lr_scheduler = deepspeed.initialize(**kwargs)
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/deepspeed/__init__.py", line 165, in initialize
x1000c0s6b0n0:     engine = DeepSpeedEngine(args=args,
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 309, in __init__
x1000c0s6b0n0:     self._configure_optimizer(optimizer, model_parameters)
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 1173, in _configure_optimizer
x1000c0s6b0n0:     basic_optimizer = self._configure_basic_optimizer(model_parameters)
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 1229, in _configure_basic_optimizer
x1000c0s6b0n0:     optimizer = DeepSpeedCPUAdam(model_parameters,
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/deepspeed/ops/adam/cpu_adam.py", line 94, in __init__
x1000c0s6b0n0:     self.ds_opt_adam = CPUAdamBuilder().load()
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/deepspeed/ops/op_builder/builder.py", line 454, in load
x1000c0s6b0n0:     return self.jit_load(verbose)
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/deepspeed/ops/op_builder/builder.py", line 497, in jit_load
x1000c0s6b0n0:     op_module = load(name=self.name,
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1284, in load
x1000c0s6b0n0:     return _jit_compile(
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1508, in _jit_compile
x1000c0s6b0n0:     _write_ninja_file_and_build_library(
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1592, in _write_ninja_file_and_build_library
x1000c0s6b0n0:     verify_ninja_availability()
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1648, in verify_ninja_availability
x1000c0s6b0n0:     raise RuntimeError("Ninja is required to load C++ extensions")
x1000c0s6b0n0: RuntimeError: Ninja is required to load C++ extensions
x1000c1s1b0n0: Loading extension module cpu_adam...
x1000c1s1b0n0: Time to load cpu_adam op: 0.502467155456543 seconds
x1000c1s1b0n0: Exception ignored in: <function DeepSpeedCPUAdam.__del__ at 0x7f42085dca60>
x1000c1s1b0n0: Traceback (most recent call last):
x1000c1s1b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/deepspeed/ops/adam/cpu_adam.py", line 102, in __del__
x1000c1s1b0n0:     self.ds_opt_adam.destroy_adam(self.opt_id)
x1000c1s1b0n0: AttributeError: 'DeepSpeedCPUAdam' object has no attribute 'ds_opt_adam'
x1000c0s6b0n0:  [WARNING]  cpu_adam cuda is missing or is incompatible with installed torch, only cpu ops can be compiled!
x1000c0s6b0n0: Using /home/users/industry/dso/lannliat/.cache/torch_extensions/py310_cu116 as PyTorch extensions root...
x1000c0s6b0n0: Traceback (most recent call last):
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/transformers/examples/pytorch/language-modeling/run_clm.py", line 634, in <module>
x1000c0s6b0n0:     main()
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/transformers/examples/pytorch/language-modeling/run_clm.py", line 582, in main
x1000c0s6b0n0:     train_result = trainer.train(resume_from_checkpoint=checkpoint)
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/transformers/src/transformers/trainer.py", line 1539, in train
x1000c0s6b0n0:     return inner_training_loop(
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/transformers/src/transformers/trainer.py", line 1659, in _inner_training_loop
x1000c0s6b0n0:     model, self.optimizer, self.lr_scheduler = self.accelerator.prepare(
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/accelerate/accelerator.py", line 1198, in prepare
x1000c0s6b0n0:     result = self._prepare_deepspeed(*args)
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/accelerate/accelerator.py", line 1537, in _prepare_deepspeed
x1000c0s6b0n0:     engine, optimizer, _, lr_scheduler = deepspeed.initialize(**kwargs)
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/deepspeed/__init__.py", line 165, in initialize
x1000c0s6b0n0:     engine = DeepSpeedEngine(args=args,
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 309, in __init__
x1000c0s6b0n0:     self._configure_optimizer(optimizer, model_parameters)
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 1173, in _configure_optimizer
x1000c0s6b0n0:     basic_optimizer = self._configure_basic_optimizer(model_parameters)
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 1229, in _configure_basic_optimizer
x1000c0s6b0n0:     optimizer = DeepSpeedCPUAdam(model_parameters,
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/deepspeed/ops/adam/cpu_adam.py", line 94, in __init__
x1000c0s6b0n0:     self.ds_opt_adam = CPUAdamBuilder().load()
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/deepspeed/ops/op_builder/builder.py", line 454, in load
x1000c0s6b0n0:     return self.jit_load(verbose)
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/deepspeed/ops/op_builder/builder.py", line 497, in jit_load
x1000c0s6b0n0:     op_module = load(name=self.name,
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1284, in load
x1000c0s6b0n0:     return _jit_compile(
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1508, in _jit_compile
x1000c0s6b0n0:     _write_ninja_file_and_build_library(
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1592, in _write_ninja_file_and_build_library
x1000c0s6b0n0:     verify_ninja_availability()
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1648, in verify_ninja_availability
x1000c0s6b0n0:     raise RuntimeError("Ninja is required to load C++ extensions")
x1000c0s6b0n0: RuntimeError: Ninja is required to load C++ extensions
x1000c0s6b0n0: Exception ignored in: <function DeepSpeedCPUAdam.__del__ at 0x7eff0daaca60>
x1000c0s6b0n0: Traceback (most recent call last):
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/deepspeed/ops/adam/cpu_adam.py", line 102, in __del__
x1000c0s6b0n0:     self.ds_opt_adam.destroy_adam(self.opt_id)
x1000c0s6b0n0: AttributeError: 'DeepSpeedCPUAdam' object has no attribute 'ds_opt_adam'
x1000c0s6b0n0: Exception ignored in: <function DeepSpeedCPUAdam.__del__ at 0x7f0145cb8a60>
x1000c0s6b0n0: Traceback (most recent call last):
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/deepspeed/ops/adam/cpu_adam.py", line 102, in __del__
x1000c0s6b0n0:     self.ds_opt_adam.destroy_adam(self.opt_id)
x1000c0s6b0n0: AttributeError: 'DeepSpeedCPUAdam' object has no attribute 'ds_opt_adam'
x1000c1s1b0n0: [2023-07-16 11:24:04,470] [INFO] [launch.py:315:sigkill_handler] Killing subprocess 2431494
x1000c0s6b0n0: [2023-07-16 11:24:04,784] [INFO] [launch.py:315:sigkill_handler] Killing subprocess 665498
x1000c0s6b0n0: [2023-07-16 11:24:04,804] [INFO] [launch.py:315:sigkill_handler] Killing subprocess 665499
```

When I tried single node training, the training procedes smoothly:
```
deepspeed --num_gpus=2 $ROOT_DIR/transformers/examples/pytorch/language-modeling/run_clm.py \
    --model_name_or_path gpt2 \
    --dataset_name wikitext \
    --dataset_config_name wikitext-2-raw-v1 \
    --per_device_train_batch_size 8 \
    --per_device_eval_batch_size 8 \
    --do_train \
    --do_eval \
    --output_dir ~/scratch/finetuned_models/debug \
    --deepspeed "$ROOT_DIR/FastChat/fastchat/ds_config/ds_config_zero3.json"
```

### Expected behavior

Expected training to proceed smoothly as in single node case.
For the error log

```bash
x1000c0s6b0n0:   File "/home/users/industry/dso/lannliat/.conda/envs/debug/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 1229, in _configure_basic_optimizer
x1000c0s6b0n0:     optimizer = DeepSpeedCPUAdam(model_parameters,
```

I believe it's best to open an issue on [DeepSpeed GitHub issues](https://github.com/microsoft/DeepSpeed/issues) page for this.

It's likely a deepspeed version issue.