Motivated by this pressing need for more efficient LLMs, this survey aims to provide a comprehensive and up-to-date
understanding of the subject. For the context of this paper, "efficiency" is defined as the optimization of computational
and memory resources without compromising model performance. Adopting a holistic approach, we explore multiple
dimensions of efficiency that are crucial for the end-to-end development of LLMs. These dimensions encompass data
utilization, architectural designs, training and tuning strategies, and inference techniques, —essentially covering the entire
pipeline of model development from algorithmic and software perspective.2 While there are existing surveys that focus
on specific aspects of LLM efficiency, such as data [316], training [241, 333, 342], tuning [323], or inference [295, 339], they
often do not provide a comprehensive view. Other works, like [267], have offered valuable insights into various efficiency
TheEfficiencySpectrumofLargeLanguageModels-AnAlgorithmicSurvey