Relative Positional Encoding with Decay Functions. Another trend is to employ trainable relative positional encodings
(RPE) that use decaying functions. This approach, exemplified by models like ALiBi [207], KERPLE [48], and Sandwich [49],
aims to focus the model’s attention predominantly on neighboring tokens. The use of decaying functions in these methods
ensures that the attention diminishes as the distance between tokens increases. ALiBi introduces a linear decaying function
to model the relationship between tokens, particularly effective for capturing the diminishing relevance of tokens as their
distance increases. KERPLE [48] uses two variations of conditionally positive definite (CPD) kernels: a logarithmic variant
and a power variant. These sophisticated kernels decay the connection between two tokens during RPE computation to
adaptively model the decreasing significance of distant token relationships. Sandwich [49], meanwhile, adopts a series of
TheEfficiencySpectrumofLargeLanguageModels-AnAlgorithmicSurvey