Deepspeed OOM When training 7B model on V100 16GB (2)
### System Info

 - Python version 3.8
- transformers - installed from source latest

**Describe the bug**
OOM When training 7B model on V100 16GB (2) with zero stage 2 and CPU offloading even when memory estimation showed far less per GPU mem requirement. 
```
-- memory estimation--
DEVICES ['Tesla V100-PCIE-16GB', 'Tesla V100-PCIE-16GB']
-------------ZERO 2------------
Estimated memory needed for params, optim states and gradients for a:
HW: Setup with 1 node, 2 GPUs per node.
SW: Model with 6650M total params.
  per CPU  |  per GPU |   Options
  148.66GB |  12.39GB | offload_optimizer=cpu
   74.33GB |  74.33GB | offload_optimizer=none
```

**Screenshots**
nvidia-smi during run
<img width="648" alt="Screenshot 2023-06-19 at 5 30 41 PM" src="https://github.com/microsoft/DeepSpeed/assets/25312635/47a512c6-b509-49c2-b111-8f7e9dac8532">

RAM usage
<img width="723" alt="Screenshot 2023-06-19 at 5 55 08 PM" src="https://github.com/microsoft/DeepSpeed/assets/25312635/e7e97e07-7203-42a7-8b89-468c2de35546">
Can see free RAM available.

**System info (please complete the following information):**
 - OS: CentOS Linux
 - GPU count and types : V100 16B X 2 single node



### Who can help?

@pacman100 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Steps to reproduce the behavior:
My code is [here](https://github.com/explodinggradients/Funtuner/blob/main/funtuner/trainer.py)
Run deepspeed funtuner/trainer.py

export PYTHONPATH="${PYTHONPATH}:/your-path/Funtuner"
please change the log_dir to your folder [here](https://github.com/explodinggradients/Funtuner/blob/c4e66209d5ee276a7eb8caf582435f1eaafbf18f/funtuner/config/config.yaml#L4) also you might want to set log_wandb=False
`dev-train` branch
 

### Expected behavior

Run w/o OOM error.
