Towards Data Science, Deep Learning, Machine Learning, Neural Networks, Naturallanguageprocessing.
multiple ways of defining these positional embeddings. For example in the original paper — Attention is All You Need, the authors used an alternating sine and cosine function to define the embeddings, as shown in figure 5. Figure 5. Positional embedding used in the original paper ( Source: image