Llm, Transformers, Fine Tuning, Gpu, Scaling Law.
paper, detailing various methods for the scaling law of training compute-optimal LLMs, that are different than OpenAI’s. They have conducted over 400 LLMs training and found out that for compute-optimal training, the model size and dataset size (number of tokens) should be scaled equally. They