'DummyOptim' object has no attribute 'step'
### System Info

- `transformers` version: 4.31.0.dev0
- Platform: Linux-5.4.0-124-generic-x86_64-with-glibc2.31
- Python version: 3.11.3
- Huggingface_hub version: 0.15.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: yes 4*A100 80GB

### Who can help?

_No response_

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

I am trying to train a model from the given [script](https://github.com/salesforce/CodeT5/blob/main/CodeT5%2B/instruct_tune_codet5p.py) in a single node multi-GPU setting 
 with DeepSpeed integration and am getting the error as given below
To reproduce one can download the script from [here](https://github.com/salesforce/CodeT5/blob/main/CodeT5%2B) along with the config file and try to run it with the [CodeAlpaca dataset](https://raw.githubusercontent.com/sahil280114/codealpaca/master/data/code_alpaca_20k.json)

```
{'batch_size_per_replica': 1,
 'cache_data': 'cache_data/instructions',
 'data_num': -1,
 'deepspeed': 'deepspeed_config.json',
 'epochs': 3,
 'fp16': False,
 'grad_acc_steps': 16,
 'instruct_data_path': 'code_alpaca_20k.json',
 'load': 'codet5p-16b',
 'local_rank': -1,
 'log_freq': 10,
 'lr': 2e-05,
 'lr_warmup_steps': 30,
 'max_len': 512,
 'save_dir': 'saved_models/instruct_codet5p_16b',
 'save_freq': 500}
  ==> Loaded 20022 samples
Loading checkpoint shards: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:25<00:00,  5.15s/it]
  ==> Loaded model from codet5p-16b, model size 16493680640
Para before freezing: 16493680640, trainable para: 16494M
Para after freezing: 16493680640, trainable para: 462M
Starting main loop
  0%|                                                                                                                                                                                                                                         | 0/936 [00:00<?, ?it/s]/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.
  warnings.warn('Was asked to gather along dimension 0, but all '

Traceback (most recent call last):
  File "/root/Custom-LLM/CodeT5/CodeT5+/instruct_tune_codet5p.py", line 212, in <module>
    main(args)
  File "/root/Custom-LLM/CodeT5/CodeT5+/instruct_tune_codet5p.py", line 181, in main
    run_training(args, model, train_data)
  File "/root/Custom-LLM/CodeT5/CodeT5+/instruct_tune_codet5p.py", line 93, in run_training
    trainer.train()
  File "/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/transformers/trainer.py", line 1537, in train
    return inner_training_loop(
           ^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/transformers/trainer.py", line 1881, in _inner_training_loop
    self.optimizer.step()
    ^^^^^^^^^^^^^^^^^^^
AttributeError: 'DummyOptim' object has no attribute 'step'
```

The same script works completely fine in a single-GPU setting. But when I switch to a multi-GPU setup I get this error

### Expected behavior

Expect the training to proceed using Deepspeed as in the case of a single-GPU set up
cc @pacman100 <|||||>Hello, please provide a minimal reproducible example that we can directly run. Providing links to scripts and dataset doesn't help and is very involved and time taking. <|||||>Also, please provide the accelerate and DeepSpeed versions, the launch command for the minimal example and the minimal example as mentioned above.<|||||>Code:
```
"""
Finetune CodeT5+ models on instruction tuning data
You can customize your own training data by following the HF dataset format to cache it to args.cache_data
Author: Yue Wang
Date: June 2023
"""

import os
import pprint
import argparse
import numpy as np
import copy
import torch
from datasets import load_dataset, load_from_disk
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer, TrainingArguments, Trainer

PROMPT_DICT = {
    "prompt_input": (
        "Below is an instruction that describes a task, paired with an input that provides further context. "
        "Write a response that appropriately completes the request.\n\n"
        "### Instruction:\n{instruction}\n\n### Input:\n{input}\n\n### Response:"
    ),
    "prompt_no_input": (
        "Below is an instruction that describes a task. "
        "Write a response that appropriately completes the request.\n\n"
        "### Instruction:\n{instruction}\n\n### Response:"
    ),
}


def get_model_size(model):
    model_parameters = filter(lambda p: p.requires_grad, model.parameters())
    model_size = sum([np.prod(p.size()) for p in model_parameters])
    return "{}M".format(round(model_size / 1e+6))


def freeze_decoder_except_xattn_codegen(model):
    print(f'Para before freezing: {model.num_parameters()}, trainable para: {get_model_size(model)}')
    for param in model.decoder.parameters():
        param.requires_grad = False

    num_decoder_layers = model.decoder.config.num_layers
    for i in range(num_decoder_layers):
        each_decoder_layer = model.decoder.transformer.h[i]
        if hasattr(each_decoder_layer, 'crossattention'):
            for param in each_decoder_layer.crossattention.parameters():
                param.requires_grad = True
            each_decoder_layer.crossattention.to(torch.float32)

        if hasattr(each_decoder_layer, 'alpha_xattn'):
            each_decoder_layer.alpha_xattn.requires_grad = True
    print(f'Para after freezing: {model.num_parameters()}, trainable para: {get_model_size(model)}')


def run_training(args, model, train_data):
    print(f"Starting main loop")

    training_args = TrainingArguments(
        #report_to='tensorboard',
        output_dir=args.save_dir,
        overwrite_output_dir=False,

        do_train=True,
        save_strategy='epoch',

        num_train_epochs=args.epochs,
        per_device_train_batch_size=args.batch_size_per_replica,
        gradient_accumulation_steps=args.grad_acc_steps,

        learning_rate=args.lr,
        weight_decay=0.0,
        warmup_steps=args.lr_warmup_steps,

        logging_dir=args.save_dir,
        logging_first_step=True,
        logging_steps=args.log_freq,
        save_total_limit=2,

        dataloader_drop_last=True,
        dataloader_num_workers=4,

        local_rank=args.local_rank,
        deepspeed=args.deepspeed,
        fp16=args.fp16,
    )

    trainer = Trainer(
        model=model,
        args=training_args,
        train_dataset=train_data,
    )

    trainer.train()

    if args.local_rank in [0, -1]:
        final_checkpoint_dir = os.path.join(args.save_dir, "final_checkpoint")
        model.save_pretrained(final_checkpoint_dir)
        print(f'  ==> Finish training and save to {final_checkpoint_dir}')


def load_tokenize_data(args):
    # Load and tokenize data
    if os.path.exists(args.cache_data):
        train_data = load_from_disk(args.cache_data)
        print(f'  ==> Loaded {len(train_data)} samples')
        return train_data
    else:
        datasets = load_dataset('json', data_files=args.instruct_data_path)['train']
        tokenizer = AutoTokenizer.from_pretrained(args.load)

        def preprocess_function(examples):
            prompt_input, prompt_no_input = PROMPT_DICT["prompt_input"], PROMPT_DICT["prompt_no_input"]
            source = [prompt_input.format_map({'instruction': instruct, 'input': inp}) if inp != ''
                      else prompt_no_input.format_map({'instruction': instruct})
                      for instruct, inp in zip(examples["instruction"], examples["input"])]
            target = [src + output + tokenizer.eos_token for src, output in zip(source, examples["output"])]

            model_inputs = tokenizer(source, max_length=args.max_len, padding="max_length", truncation=True)
            labels = tokenizer(target, max_length=args.max_len, padding="max_length", truncation=True)
            model_inputs["decoder_input_ids"] = copy.deepcopy(labels["input_ids"])

            # changing labels: convert all tokens in the duplicate prefix prompt and the padding part to -100
            eos_token_id = tokenizer.eos_token_id
            for x, y in zip(model_inputs["input_ids"], labels["input_ids"]):
                label_prefix_len = x.index(eos_token_id) if eos_token_id in x else len(x)
                y[:label_prefix_len] = [-100] * label_prefix_len

                if eos_token_id in y:
                    pad_len = len(y) - y.index(eos_token_id) - 1
                    if pad_len > 0:
                        y[y.index(eos_token_id) + 1:] = [-100] * pad_len

            # shift labels to the right as the decoder input and add decoder start token id
            decoder_start_id = tokenizer.eos_token_id
            for z in model_inputs["decoder_input_ids"]:
                z[1:] = z[:-1]
                z[0] = decoder_start_id

            model_inputs["labels"] = copy.deepcopy(labels["input_ids"])
            model_inputs["decoder_attention_mask"] = labels["attention_mask"]
            return model_inputs

        train_data = datasets.map(
            preprocess_function,
            batched=True,
            remove_columns=datasets.column_names,
            num_proc=64,
            load_from_cache_file=False,
        )

        print(f'  ==> Loaded {len(train_data)} samples')
        train_data.save_to_disk(args.cache_data)
        print(f'  ==> Saved to {args.cache_data}')
        return train_data


def main(args):
    argsdict = vars(args)
    print(pprint.pformat(argsdict))

    # Save command to file
    with open(os.path.join(args.save_dir, "command.txt"), 'w') as f:
        f.write(pprint.pformat(argsdict))

    # Load and tokenize data using the tokenizer from `args.load`. If the data is already cached, load it from there.
    # You can customize this function to load your own data for any Seq2Seq LM tasks.
    train_data = load_tokenize_data(args)

    if args.data_num != -1:
        train_data = train_data.select([i for i in range(args.data_num)])

    # Load model from `args.load`
    model = AutoModelForSeq2SeqLM.from_pretrained(args.load, torch_dtype=torch.float16,
                                                  low_cpu_mem_usage=True, trust_remote_code=True)

    print(f"  ==> Loaded model from {args.load}, model size {model.num_parameters()}")
    #freeze_decoder_except_xattn_codegen(model)

    run_training(args, model, train_data)


if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="CodeT5+ instruction tuning")
    parser.add_argument('--data-num', default=-1, type=int)
    parser.add_argument('--max-len', default=512, type=int)
    parser.add_argument('--instruct-data-path', default='code_alpaca_20k.json', type=str)
    parser.add_argument('--cache-data', default='cache_data/instructions', type=str)
    parser.add_argument('--load', default='Salesforce/codet5p-16b', type=str)

    # Training
    parser.add_argument('--epochs', default=3, type=int)
    parser.add_argument('--lr', default=2e-5, type=float)
    parser.add_argument('--lr-warmup-steps', default=30, type=int)
    parser.add_argument('--batch-size-per-replica', default=1, type=int)
    parser.add_argument('--grad-acc-steps', default=16, type=int)
    parser.add_argument('--local_rank', default=-1, type=int)
    parser.add_argument('--deepspeed', default=None, type=str)
    parser.add_argument('--fp16', default=False, action='store_true')

    # Logging and stuff
    parser.add_argument('--save-dir', default="saved_models/instruct_codet5p_16b", type=str)
    parser.add_argument('--log-freq', default=10, type=int)
    parser.add_argument('--save-freq', default=500, type=int)

    args = parser.parse_args()

    os.makedirs(args.save_dir, exist_ok=True)

    main(args)
```

command:
```
deepspeed CodeT5+/instruct_tune_codet5p.py --load $MODEL --save-dir $SAVE_DIR --instruct-data-path code_alpaca_2k.json --fp16 --deepspeed ~/transformers/tests/deepspeed/ds_config_zero3.json
```

Output:
![Screenshot 2023-07-04 at 4 44 34 PM](https://github.com/huggingface/transformers/assets/13534540/cb880856-433f-4f2a-b5e1-40b91c845566)

- `Accelerate` version: 0.21.0.dev0
- Platform: Linux-5.4.0-125-generic-x86_64-with-glibc2.31
- Python version: 3.10.11
- Numpy version: 1.24.4
- PyTorch version (GPU?): 2.0.1 (True)
- PyTorch XPU available: False
- System RAM: 503.55 GB
- GPU type: NVIDIA A100-SXM4-80GB
- `Accelerate` default config:
	Not found
- 
- `transformers` version: 4.31.0.dev0
- Platform: Linux-5.4.0-125-generic-x86_64-with-glibc2.31
- Python version: 3.10.11
- Huggingface_hub version: 0.15.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

```bash
--------------------------------------------------
DeepSpeed C++/CUDA extension op report
--------------------------------------------------
NOTE: Ops not installed will be just-in-time (JIT) compiled at
      runtime if needed. Op compatibility means that your system
      meet the required dependencies to JIT install the op.
--------------------------------------------------
JIT compiled ops requires ninja
ninja .................. [OKAY]
--------------------------------------------------
op name ................ installed .. compatible
--------------------------------------------------
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
async_io ............... [NO] ....... [NO]
cpu_adagrad ............ [NO] ....... [OKAY]
cpu_adam ............... [NO] ....... [OKAY]
fused_adam ............. [NO] ....... [OKAY]
fused_lamb ............. [NO] ....... [OKAY]
quantizer .............. [NO] ....... [OKAY]
random_ltd ............. [NO] ....... [OKAY]
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.0
 [WARNING]  using untested triton version (2.0.0), only 1.0.0 is known to be compatible
sparse_attn ............ [NO] ....... [NO]
spatial_inference ...... [NO] ....... [OKAY]
transformer ............ [NO] ....... [OKAY]
stochastic_transformer . [NO] ....... [OKAY]
transformer_inference .. [NO] ....... [OKAY]
--------------------------------------------------
DeepSpeed general environment info:
torch install path ............... ['/home/sourab/miniconda3/envs/hf/lib/python3.10/site-packages/torch']
torch version .................... 2.0.1
deepspeed install path ........... ['/home/sourab/miniconda3/envs/hf/lib/python3.10/site-packages/deepspeed']
deepspeed info ................... 0.9.5, unknown, unknown
torch cuda version ............... 11.8
torch hip version ................ None
nvcc version ..................... 11.8
deepspeed wheel compiled w. ...... torch 2.0, cuda 11.8
```

Therefore, unable to reproduce your error.  <|||||>I see similar issue. <|||||>I'm also having this issue!

I believe it happens when you include an `"optimizer"` as part of your Deepspeed config. E.g.

```json
"optimizer": {
    "type": "AdamW",
    "params": {
         "lr": "auto",
         "betas": "auto",
         "eps": "auto",
         "weight_decay": "auto",
     },
}
```

Then, `optimizer = DummyOptim(params=model_parameters)`.

https://github.com/huggingface/transformers/blob/e42587f596181396e1c4b63660abf0c736b10dae/src/transformers/deepspeed.py#L282-L288

And eventually `Trainer` tries to call `optimizer.step()`. However, `DummyOptim` doesn't have a `step()` function.

https://github.com/huggingface/accelerate/blob/8514c35192ac9762920f1ab052e5cea4c0e46eeb/src/accelerate/utils/deepspeed.py#L226-L246

I'm not sure what the appropriate fix is: should we not use `DummyOptim` at all or add a `step()` function (that does nothing?) to it?

P.S. the same problem applies to `"scheduler"`.

https://github.com/huggingface/transformers/blob/e42587f596181396e1c4b63660abf0c736b10dae/src/transformers/deepspeed.py#L303-L304

Please take a look, @pacman100 -- thanks!<|||||>@apoorvkh , post the `accelerator.prepare` they should be replaced with correct optimizer and scheduler from DeepSpeed and hence should not result in any issues. As shown above, unable to reproduce it. A minimal way to reproduce it would help me deep dive<|||||>See https://github.com/huggingface/transformers/blob/main/src/transformers/trainer.py#L1653-L1656 wherein it `accelerator.prepare` internally calls `deepspeed.initialize` and replace the dummy objects with appropriate  ones returned by DeepSpeed.<|||||>Thanks, that was helpful! Using that information, we found that the cause was because `ACCELERATE_USE_DEEPSPEED=true` was not already being set internally. (If you set it manually, everything works as expected.)

We unfortunately can't very easily share a minimal reproducible example, but will continue to debug soon.