Machine Learning, AI, Deep Learning, NLP, Llm.
are easy to fine-tune and also have less latency at inference. These models should not be to their lowest possible loss to be compute optimal. Current LLMs The main question for their research is “Given a fixed FLOPs budget, how should one trade-off model size and the number of training tokens?”.