Model Size. Model size, defined as the disk space required for storing the entire model, is often the initial consideration when
training a new LLM or working with a pre-trained model. Given that exceedingly large models may be infeasible to store or
run, this metric is particularly crucial for real-world deployments, especially in storage-constrained environments like edge
devices. Expressed in units such as gigabytes (GB) or megabytes (MB), model size is influenced by several factors. While the
number of parameters plays a significant role, other elements like the data type used for parameters (e.g., float16, int8) and
specific architectural choices also contribute. In addition to its direct impact on storage requirements, model size serves as an
indirect indicator of the computational resources needed for both training and inference.
TheEfficiencySpectrumofLargeLanguageModels-AnAlgorithmicSurvey