Llm, Transformers, Fine Tuning, Gpu, Scaling Law.
Computation Speed Measured in FLOPS (Floating point operations per second) FLOPS, or Floating point operations per second is the speed of compute. Below is the table of Nvidia A100 and H100 GPUs performance, as shown in Fig8. One can estimate number of GPUs required to train LLMs based on the table