21

Efficient LLM Algorithmic Survey, Nov, 2023, USA.

Ding, Chen, et al.

streamlined versions. Knowledge distillation involves transferring insights from larger, more complex ’teacher’ models
to smaller, more efficient ’student’ models, helping to maintain high performance even in reduced models. Quantization
reduces computational load and storage requirements by representing floating-point numbers in LLMs with fewer bits.
Low-rank decomposition approximates the heavy weight matrices in LLMs through low-rank matrices, further economizing
computational resources. It’s worth noting that some of these approaches require specialized computing libraries and hardware
to achieve realistic resource savings and speed-up.

7.2 Pruning
Pruning techniques aim at identifying the redundancy inside operators of LLMs. Existing pruning techniques can be broadly
classified as unstructured, semi-structured, and structured pruning.
TheEfficiencySpectrumofLargeLanguageModels-AnAlgorithmicSurvey