GPU implementation (except LAMB).
Optimizer
DeepSpeed's main optimizers are Adam, AdamW, OneBitAdam, and Lamb. These have been thoroughly tested with ZeRO and are
thus recommended to be used. It, however, can import other optimizers from torch. The full documentation is here.
If you don't configure the optimizer entry in the configuration file, the [Trainer] will
automatically set it to AdamW and will use the supplied values or the defaults for the following command line
arguments: --learning_rate, --adam_beta1, --adam_beta2, --adam_epsilon and --weight_decay.
Here is an example of the auto-configured optimizer entry for AdamW:
json
{
   "optimizer": {
       "type": "AdamW",
       "params": {
         "lr": "auto",
         "betas": "auto",
         "eps": "auto",
         "weight_decay": "auto"
