Machine Learning, AI, Deep Learning, NLP, Llm.
approach, they vary the model size for a fixed set of 9 different training FLOP counts (ranging from 10^18 to 10^21 FLOPs). This approach answers the question “For a given FLOP budget, what is the optimal parameter count?”. While training they suggest that for a model with over 𝐷 tokens, a cosine