Positional and Segment Embeddings in BERT
#### Question

I'm trying to train a BERT language model from scratch using the huggingface library. I'm not sure I completely understand how positional and segment embeddings work in BERT.  The original BERT paper states that unlike transformers, positional and segment embeddings are learned. What exactly does this mean? 

How do positional embeddings help in predicting masked tokens? Is the positional embedding of the masked token predicted along with the word?

How has this been implemented in the huggingface library? 
They are not predicted, they are learnt alongside the token embedding themselves. They work exactly the same w.r.t. to training: they are all simply embeddings but with a different vocab size.

https://github.com/huggingface/transformers/blob/9a473f1e43221348334b9e7f95bb45770b7ef268/src/transformers/modeling_bert.py#L154-L156

The output of all three embeddings are summed up before passing them to the transformer layers.

Positional embeddings can help because they basically highlight the position of a word in the sentence. A word in the first position likely has another meaning/function than the last one. Also, the same word likely will have a different syntactic function in the first vs. last position. Positional embeddings thus play some kind of syntactic role where they tell the model that a word can have a different meaning/syntactic function depending on its position.<|||||>Thank you for the explanation. What is the benefit of learning positional embeddings as opposed to the technique adopted by transformers where the positional encoding is directly added to each input embedding and not learnt?<|||||>When is a positional embedding directly added? An embedding is an embedding and as such always needs to be learned or transfered. <|||||>Section 3.5 of the paper '[Attention is All You Need](https://arxiv.org/pdf/1706.03762.pdf)' explains the positional encoding in the case of transformers. They use _'sine and cosine functions of different frequencies'_ to inject information about the position of the tokens. Learned positional embeddings do not seem to help in the case of the original transformers. BERT on the other hand 'learns' positional embeddings. I wanted to understand the benefits of learning these positional embeddings in BERT.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
