Llm Pre Training, Llm Architecture, Model Quantization, Scaling Model Training, Llm Domain Adaptation.
https://www.coursera.org/learn/generative-ai-with-llms Scaling model training across multiple GPUs using techniques like DDP and FSDP brings notable efficiency and performance benefits. DDP allows parallel processing on multiple GPUs when the model fits onto a single GPU, while FSDP with ZeRO