Llm Pre Training, Llm Architecture, Model Quantization, Scaling Model Training, Llm Domain Adaptation.
number of GPUs. Source : https://www.coursera.org/learn/generative-ai-with-llms Efficient Training with FSDP FSDP distributes data and shards model parameters, gradients, and optimizer states across GPUs. Unlike DDP, FSDP requires collecting data from all GPUs before the forward and backward pass.