Llm, Transformers, Fine Tuning, Gpu, Scaling Law.
requirement (C )is ~6 ND or 8 ND for distributed training, where N is the number of parameters of the transformers, and D is the training data size, measured in tokens. C forward is ~ 2 ND; C backward is ~4 ND. Fig7: PFLOPs are growing to catch up with the scaling law of LLMs Appendix 2: