Llm Pre Training, Llm Architecture, Model Quantization, Scaling Model Training, Llm Domain Adaptation.
https://www.coursera.org/learn/generative-ai-with-llms FSDP, inspired by the ZeRO technique, provides a solution when the model is too large to fit in the memory of a single GPU. ZeRO (Zero Redundancy Optimizer) aims to optimize memory usage by distributing or sharding model parameters, gradients,