I’ve implemented a transformer model following along with Peter Bloem’s blog 52
I find myself confused by the high level meaning of the position embeddings. When I look at papers/articles describing position embeddings, they all seem to indicate we embed the positions in individual sentences, which makes sense.
But if you look at the code accompanying Peter Bloem’s blog, it seems the position embeddings are for the entire sequence (i.e., potentially many sentences). The position embedding layer is defined as nn.Embedding(a, b) where a equals the dimension of the word embedding vectors, and b is set to the length of the longest sequence (I believe 512).
Does this mean we are creating position vectors for 512 different positions? If so, I feel like that doesn’t make sense. The first word of a sentence could be at position 1 and position 242 in one case, but in another case position 242 could be the last word in a sentence (or any word).
I used the same style of position embedding as did Bloem - that is my position embedding layer is nn.Embedding(word_embedding_size, len_longest_sequence). I am getting good results, yet I feel quite confused about the position embeddings.