Llm, Transformers, Fine Tuning, Gpu, Scaling Law.
Core with sparse, A100 will deliver 624TFLOPS (6.24 x 10¹⁴ floating operation per second). It is reported that OpenAI used 25,000 A100s. Hence, 1.33x10²⁶/(6.24x10¹⁴ x 25,000) = 8525641 seconds or 98.67 days, which sort of aligns with the report and post. Good, the math works out! :) Fig8: Table of