your needs.
In your own programs, you can also use the following approach if you'd like to modify the DeepSpeed config as a master
and configure [TrainingArguments] based on that. The steps are:
Create or load the DeepSpeed configuration to be used as a master configuration
Create the [TrainingArguments] object based on these values
Do note that some values, such as scheduler.params.total_num_steps are calculated by
[Trainer] during train, but you can of course do the math yourself.
ZeRO
Zero Redundancy Optimizer (ZeRO) is the workhorse of DeepSpeed. It
support 3 different levels (stages) of optimization. The first one is not quite interesting for scalability purposes,
therefore this document focuses on stages 2 and 3. Stage 3 is further improved by the latest addition of ZeRO-Infinity.
You will find more indepth information in the DeepSpeed documentation.
