You may also try the ZeRO-3 with CPU and NVMe offload as explained further in this document.
- TODO: Benchmark whether we can get better performance out of ZeRO-3 vs. ZeRO-2 on a single GPU, and then
recommend ZeRO-3 config as starting one. 
Notes:
if you need to run on a specific GPU, which is different from GPU 0, you can't use CUDA_VISIBLE_DEVICES to limit
  the visible scope of available GPUs. Instead, you have to use the following syntax:
bash
  deepspeed --include localhost:1 examples/pytorch/translation/run_translation.py ...
In this example, we tell DeepSpeed to use GPU 1 (second gpu).
Deployment in Notebooks
The problem with running notebook cells as a script is that there is no normal deepspeed launcher to rely on, so
under certain setups we have to emulate it.
If you're using only 1 GPU, here is how you'd have to adjust your training code in the notebook to use DeepSpeed.
