Llm, Transformers, Fine Tuning, Gpu, Scaling Law.
model sizes, dataset sizes, and other factors such as: Model Size (N): Ranges from 768 to 1.5 billion non-embedding parameters. Dataset Size (D): Varies from 22 million to 23 billion tokens. Model shape: Includes variations in depth, width, number of attention heads, and feed-forward dimension.