BertEmbeddings code for position_embeddings and word_embeddings
   I am trying to figure out how exactly positions and words are embedded in Bert.  I checked the code, and it seems following code is used for the purpose.   This implies that both the embedding function are calling same `__init__' function from the class `nn.Embedding`.  Where exactly is the code doing the actual position embedding (sine and cosine based). Where can I see that code ?.  

    self.word_embeddings = nn.Embedding(config.vocab_size, config.hidden_size, padding_idx=config.pad_token_id)
    self.position_embeddings = nn.Embedding(config.max_position_embeddings, config.hidden_size)
    inputs_embeds = self.word_embeddings(input_ids)
    position_embeddings = self.position_embeddings(position_ids)`
Hi, BERT uses absolute positional embeddings, not based on sinusoidal functions. The positional embeddings are learned. 

You can check the [XLNet code](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_xlnet.py#L704-L712) or the [TransfoXL code](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_transfo_xl.py#L170-L186) for examples showcasing sinusoidal (relative) embeddings.