Llm, Transformers, Fine Tuning, Gpu, Scaling Law.
complexities of a model are measured in FLOPs, or floating-point operations. They are used to estimate the amount of computational resources required to train and run a model, as shown in the y-axis of Fig7. To put it into perspective, it is estimated that OpenAI used 133 billion petaFLOPs to train