Machine Learning, AI, Deep Learning, NLP, Llm.
they are changing the number of training tokens with a fixed number of FLOPS. Using power law, they found out that the optimal model size for Gopher’s compute budget (5.76 × 10^23) is 67B and training tokens should be 1.5 trillion. Training curve envelope Approach 2: IsoFLOP profiles In the second