Llm, Transformers, Fine Tuning, Gpu, Scaling Law.
https://github.com/Lightning-AI/litgpt/blob/410a7126f82ea550d4a43dab89367547b073b5a3/litgpt/utils.py#L321 def flops_per_param(max_seq_length: int, n_layer: int, n_embd: int, n_params: int) -> int: flops_per_token = 2 * n_params # each parameter is used for a MAC (2 FLOPS) per network operation #