The cpu_offload additional option requires --fp16.
This is an area of active development, so make sure you have a source install of fairscale to use this feature as
  some bugs you encounter may have been fixed there already.
Known caveats:
This feature is incompatible with --predict_with_generate in the run_translation.py script.
Using --sharded_ddp zero_dp_3 requires wrapping each layer of the model in the special container
  FullyShardedDataParallelism of fairscale. It should be used with the option auto_wrap if you are not
  doing this yourself: --sharded_ddp "zero_dp_3 auto_wrap".
Sections that were moved:
[ DeepSpeed
| Installation
| Deployment with multiple GPUs
| Deployment with one GPU
| Deployment in Notebooks
| Configuration
| Passing Configuration
| Shared Configuration
| ZeRO
| ZeRO-2 Config
