Llm, Transformers, Fine Tuning, Gpu, Scaling Law.
indicating that models, including Gopher, GPT-3 and Megatron-Turing NLG can be trained with a lot less model parameters. (Left) Optimal number of tokens and parameters for a given FLOP budget (Right) Overlaid predictions of modified scaling law and OpenAI’s scaling law DeepMind published another