Llm, Transformers, Fine Tuning, Gpu, Scaling Law.
estimate_flops(model: "GPT", training: bool) -> int: """Measures estimated FLOPs for MFU. Refs: * https://ar5iv.labs.arxiv.org/html/2205.05198#A1 * https://ar5iv.labs.arxiv.org/html/2204.02311#A2 """ # using all parameters for this is a naive over estimation because not all model parameters