            "device": "cpu",
            "pin_memory": true
        },
        "allgather_partitions": true,
        "allgather_bucket_size": 5e8,
        "overlap_comm": true,
        "reduce_scatter": true,
        "reduce_bucket_size": 5e8,
        "contiguous_gradients": true
    }
}
Performance tuning:
enabling offload_optimizer should reduce GPU RAM usage (it requires "stage": 2)
"overlap_comm": true trades off increased GPU RAM usage to lower all-reduce latency. overlap_comm uses 4.5x
  the allgather_bucket_size and reduce_bucket_size values. So if they are set to 5e8, this requires a 9GB
  footprint (5e8 x 2Bytes x 2 x 4.5). Therefore, if you have a GPU with 8GB or less RAM, to avoid getting
  OOM-errors you will need to reduce those parameters to about 2e8, which would require 3.6GB. You will want to do
