Intuitive explanation for summing the embedding and positional encoding in the Transformer's embedding In the Transformer model, the embedding and positional encoding are summed together to represent a word in each location ('positional embedding' from now on).
This way, each cell contains semantic and positional information. However, since two types of information are represented by one number only, it seems unclear to which extent semantic information and positional information are represented in each cell.
For example:
word1 = [0.2, 0.1, -0.3]
Can be the sum of:
embedding1 = [-0.34,0.52,0.69]
positional_encoding1 = [0.54,-0.42,-0.99]

Or from:
embedding1 = [-0.64,-0.81,0.16]
positional_encoding1 = [0.84,0.91,0.14]

Hence, summing two cells makes no sense to me. Yet, Transformers are very effective algorithms, so I probably miss something. Is the problem that I describe a problem indeed? And can someone provide an intuitive explanation for summing the embedding and positional encoding?

Positional encoding is typically a deterministic function, so it's always the same on all inputs. You are adding the same vector to all of your inputs.
For example, if you were using a naive positional encoding, you would use
positional_encoding = [1, 2, 3, ..., n] for all inputs.
In Attention Is All You Need of course, they use a more sophisticated function, but it's still deterministic.
It can seem strange to add both embeddings together, but the Transformer isn't confused since the variability it sees in embedded inputs isn't caused by a blend of word and positional embeddings, it's only caused by the word embedding. Positional encoding changes every input in the same way.
