Llm Pre Training, Llm Architecture, Model Quantization, Scaling Model Training, Llm Domain Adaptation.
What is Quantization? Quantization involves reducing the memory required to store model weights by decreasing their precision. Instead of the default 32-bit floating-point numbers (FP32) used to represent parameters, quantization employs 16-bit floating-point numbers (FP16) or even 8-bit integers