Decomposition on Embedding Layer. ALBERT [145] uses factorization for the embedding layer, which is one of the
largest consumers of model parameters. Since the power of Transformer mainly comes from its contextual learning ability,
the parameters in the token embedding layer are not efficient. It intuitively makes sense to reduce them by factorizing the
embedding matrix. Self-Attentive Factorized embeddings (SAFE) [217] studied ways to share weights in transformers by
adding a small self-attention layer on the basis of linear projection to achieve better performance than the alternatives.
LightFormer [176] more effectively utilizes the parameter knowledge of the well-trained Transformer, and accelerates the
convergence of the model factorization on the embedding layers.
TheEfficiencySpectrumofLargeLanguageModels-AnAlgorithmicSurvey