Answers.Body: <p>BERT uses trained position embeddings. The <a href="https://arxiv.org/pdf/1810.04805.pdf" rel="noreferrer">original paper</a> does not say it explicitly, the term position <strong>embeddings</strong> (as opposed to encoding) suggests it is trained. When you look at BERT layers in HuggingFace Transformers, you will the dimension of the trained positions embeddings (768×512), which is also the reason why BERT cannot accept input longer than 512 tokens.</p>
  <pre><code>from transformers import AutoModel
  model = AutoModel.from_pretrained(&quot;bert-base-cased&quot;)
  print(model.embeddings)
  </code></pre>
  <pre><code>BertEmbeddings(
    (word_embeddings): Embedding(28996, 768, padding_idx=0)
    (position_embeddings): Embedding(512, 768)
    (token_type_embeddings): Embedding(2, 768)
    (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
    (dropout): Dropout(p=0.1, inplace=False)
  )
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-09-15T08:46:28.207Z
Answers.LastActivityDate: 2022-09-15T08:46:28.207Z
Answers.UserDisplayName: Jindřich
Body: <p>It is a little bit confusing that someone is explaining that BERT is using sinusoidal functions for BERT position encoding and someone is saying BERT just uses absolute position.</p>
  <p>I checked that Vaswani 2017 et al., used a sinusoidal function for position embedding, but not sure BERT used the same technique since the BERT paper does not mention it much.</p>
  <p>My question is</p>
  <ol>
  <li>Did BERT use absolute position embedding that is learnable, not with sinusoidal function?</li>
  <li>If so, is the absolute position the same dimension as the input embedding dimension for an addition?</li>
  </ol>
  
Tags: <bert><positional-encoding>
Title: Which positional encoding BERT use?
UserDisplayName: yoon
ThreadText: # Which positional encoding BERT use?
  
  (Asked by: username_0 on 2022-09-08)
  
  It is a little bit confusing that someone is explaining that BERT is using sinusoidal functions for BERT position encoding and someone is saying BERT just uses absolute position.
  
  I checked that Vaswani 2017 et al., used a sinusoidal function for position embedding, but not sure BERT used the same technique since the BERT paper does not mention it much.
  
  My question is
  
  1. Did BERT use absolute position embedding that is learnable, not with sinusoidal function?
  2. If so, is the absolute position the same dimension as the input embedding dimension for an addition?
  
  **Tags:** bert, positional-encoding
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-09-15. Score: 5
  
  BERT uses trained position embeddings. The original paper does not say it explicitly, the term position embeddings (as opposed to encoding) suggests it is trained. When you look at BERT layers in HuggingFace Transformers, you will the dimension of the trained positions embeddings (768×512), which is also the reason why BERT cannot accept input longer than 512 tokens.
  from transformers import AutoModel
  model = AutoModel.from_pretrained("bert-base-cased")
  print(model.embeddings)
  
  BertEmbeddings(
    (word_embeddings): Embedding(28996, 768, padding_idx=0)
    (position_embeddings): Embedding(512, 768)
    (token_type_embeddings): Embedding(2, 768)
    (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
    (dropout): Dropout(p=0.1, inplace=False)
  )
  
  
  ---
  
  