Llm, Transformers, Fine Tuning, Gpu, Scaling Law.
Llama 3 as an example, it’s a model with 70B parameters and 15T tokens. The loss is L (70x10⁹, 15x10¹²) = 1.86, which is a bit higher than the estimation (i.e. 1.72) using Eq1. Appendix 1: Computation Budget Measured in FLOPs (Floating point operations) The compute resources or computational