Llm, Transformers, Fine Tuning, Gpu, Scaling Law.
floating-point operations, what would be the optimal combination of model size and training data size (measured in number of tokens) that yields the lowest loss? The compute budget comes in various forms: PF-days (10¹⁵ FLOPs/second x 24 hours x 3600 seconds/hour, arriving at 8.64x10¹⁹ FLOPs) is one