Llm, Transformers, Fine Tuning, Gpu, Scaling Law.
size for a given budget is 40B parameters. Fig6: Estimated optimal training FLOPs and training tokens for various model size For optimal compute-efficient training, DeepMind suggests to have ≥ 20 training tokens for every 1 model parameter, as shown in Fig6. To put it into perspective, Llama 3