Machine Learning, AI, Deep Learning, NLP, Llm.
They tried three different approaches to answer this question. They have assumed a power-law relationship between compute and model size. Approach 1: Fix model sizes and vary number of training tokens In the first approach, they have fixed the model sizes (75M, 250M, 500M, 1B, 2.5B, 5B, 10B) and