Floating Point Operations (FLOPs). Floating-point operations (FLOPs) is commonly used to gauge the computational
complexity of LLMs. This metric counts the number of floating-point operations like addition, subtraction, multiplication, and
division, giving an estimate of the computation done during a single forward pass. While FLOPs offer valuable insights into
computational needs and potential energy use, they are not a complete measure. Other factors, such as system parallelism
and architectural choices, also play a role in determining a model’s overall computational efficiency. A higher FLOPs count
usually means the model is more computationally demanding, which can be a challenge for deployment in environments
with limited resources. As a result, optimizing this metric is often a key focus in the development of more efficient LLMs.

5

Efficient LLM Algorithmic Survey, Nov, 2023, USA.

Ding, Chen, et al.
TheEfficiencySpectrumofLargeLanguageModels-AnAlgorithmicSurvey