Question on the position embedding of DistilBERT
## ❓ Questions & Help

To the best of my knowledge, sinusoidal position embedding is used in the training procedure of DistilBERT, which is computed by [create_sinusoidal_embeddings](https://github.com/huggingface/pytorch-transformers/blob/master/pytorch_transformers/modeling_distilbert.py#L52).

When I compute the position embedding by using `create_sinusoidal_embeddings` with `n_pos=512` and `dim=768`, I got the following position embedding tensor:

```python
Parameter containing:
tensor([[ 0.0000e+00,  1.0000e+00,  0.0000e+00,  ...,  1.0000e+00,
          0.0000e+00,  1.0000e+00],
        [ 8.4147e-01,  5.4030e-01,  8.2843e-01,  ...,  1.0000e+00,
          1.0243e-04,  1.0000e+00],
        [ 9.0930e-01, -4.1615e-01,  9.2799e-01,  ...,  1.0000e+00,
          2.0486e-04,  1.0000e+00],
        ...,
        [ 6.1950e-02,  9.9808e-01,  5.3551e-01,  ...,  9.9857e-01,
          5.2112e-02,  9.9864e-01],
        [ 8.7333e-01,  4.8714e-01,  9.9957e-01,  ...,  9.9857e-01,
          5.2214e-02,  9.9864e-01],
        [ 8.8177e-01, -4.7168e-01,  5.8419e-01,  ...,  9.9856e-01,
          5.2317e-02,  9.9863e-01]])
```

However, when I looked into the position embedding from the pre-trained DistilBERT checkpoint files (`distilbert-base-uncased` and `distilbert-base-uncased-distilled-squad`,), I got the position embedding tensor as follows:

```python
tensor([[ 1.7505e-02, -2.5631e-02, -3.6642e-02,  ...,  3.3437e-05,
          6.8312e-04,  1.5441e-02],
        [ 7.7580e-03,  2.2613e-03, -1.9444e-02,  ...,  2.8910e-02,
          2.9753e-02, -5.3247e-03],
        [-1.1287e-02, -1.9644e-03, -1.1573e-02,  ...,  1.4908e-02,
          1.8741e-02, -7.3140e-03],
        ...,
        [ 1.7418e-02,  3.4903e-03, -9.5621e-03,  ...,  2.9599e-03,
          4.3435e-04, -2.6949e-02],
        [ 2.1687e-02, -6.0216e-03,  1.4736e-02,  ..., -5.6118e-03,
         -1.2590e-02, -2.8085e-02],
        [ 2.6413e-03, -2.3298e-02,  5.4922e-03,  ...,  1.7537e-02,
          2.7550e-02, -7.7656e-02]])
```

I am wondering if I missed or misunderstood something in details. Why is there a difference between these two position embedding tensors? I think the sinusoidal position embedding should be unchanged during training.

Thanks a lot!


Hello @gpengzhi 
You're right, the name is quite confusing: the second matrix of embeddings that you're showing is actually initialized from `bert-base-uncased` (compare with `bert = BertModel.from_pretrained('bert-base-uncased'); print(bert.embeddings.position_embeddings.weight)`). Once initialized, these position embeddings are frozen (both distillation or fine-tuning)
Victor