Llm Pre Training, Llm Architecture, Model Quantization, Scaling Model Training, Llm Domain Adaptation.
Stage 1 shards only optimizer states, reducing memory usage by up to a factor of four. Stage 2 shards gradients, further reducing memory usage by up to eight times when combined with Stage 1. Stage 3 shards all components, including model parameters, with memory reduction scaling linearly with the