Thank you for your reply! I’m glad I am on the right track. However, I still feel confused about the concept of position embedding for an entire sequence of sentences.
Suppose another input sequence is also length 8: c1 c2 c3 c4 c5. d1 d2 d3. Our position 5 corresponds to the last word of a sentence in my sample and the first word of the sentence in your sample. Position 5 could be the middle of a sentence for a third sample. What exactly is position 5 learning then?
The embedding vectors for “cat” vs. “dog” make sense - they are different words that can be mapped to a similar vector space (e.g., animal, four legs, etc.). But the meaning of position 5 seems random in this example. It can be early sentence, mid-sentence, or late sentence. The only deterministic thing about it seems to be that it comes after position 4 and before position 6. But positions 4 and 6 can mean different things depending on the sample (similarly, positions 254 and 256 can mean different things).
I plan to spend a lot of time studying this and related issues to come to a deeper understanding. But in the meantime I am hoping to gain some high level intuition to what the position embeddings mean. I found a paper 9 that seems to get at the question. I hope to find some insights.