Llm, Transformers, Fine Tuning, Gpu, Scaling Law.
estimated that the Nc = 8.8x10¹³, AlphaN = 0.076, AlphaD = 0.095, and Dc = 5.4x1⁰¹³. Using Llama 3 as an example, it has 70B parameters and 15T tokens. Hence, the loss is L (70x10⁹, 15x10¹²) = 1.72. Sample-Efficient LLMs Fig3: LLM’s test loss v.s. dataset size or compute by model size Large models