Transformers (Vaswani et al., 2017) incorporate position in-
formation via different positional embedding designs. The
common positional embedding design can generally be cate-
gorized into two classes: absolute position embeddings and
relative positional encodings. The absolute position embed-
ding provides the absolute positions, which embeds each
absolute position i into position vector pi and adds word
embeddings to their corresponding pi before feeding them
to the model. Examples of such include sinusoidal position
embeddings (Vaswani et al., 2017), and learned position
embeddings in GPT3 (Brown et al., 2020) and OPT (Zhang
et al., 2022), or adding the dot product between two tokens’
position embeddings on the attention logit (Ke et al., 2020).
Recently, relative positional encodings have been proposed
to instead use distance information between tokens and
have become the mainstream of position embedding. This
information is usually applied in attention layers. Exam-
Self-Extend LLM