Llm, Transformers, Fine Tuning, Gpu, Scaling Law.
applied three approaches for scaling law and overlaid their findings with OpenAI’s, as shown in the right of Fig4. They found out that in order to achieve the same model performance, all three approaches suggest much smaller model size. That is with a given compute budget and training data, one can