Llm, Transformers, Fine Tuning, Gpu, Scaling Law.
and compute requirement (i.e. the 6 ND estimate). For example, GPT4 requires 133 billion petaFLOPs, which is 1.33 x 10²⁶ FLOPs (or 1.7 trillion parameters and using 13 trillion tokens, resulting in 6 x 1.7 x 10¹² x 13 x 10¹², 1.33 x 10²⁶ FLOPs). Assuming the computation is running using FP16 Tensor