where is the position emdeddings in bert for training a new model from scratch ?
# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**: 
Right here

https://github.com/huggingface/transformers/blob/31f2437f07cf014a042789f52fa1519a485e8b2b/src/transformers/modeling_bert.py#L150

In the future, please make an effort to write a decent post that explains exactly what you need. This is very low quality.<|||||>Sry for low quality. But how about its initialization？ In the paper， it uses sin and cos functions， while the code seems like using random initialization.




| |
Wang
|
|
邮箱：zhipengwang1121@163.com
|

Signature is customized by Netease Mail Master

On 03/10/2020 22:15, Bram Vanroy wrote:

Right here

https://github.com/huggingface/transformers/blob/31f2437f07cf014a042789f52fa1519a485e8b2b/src/transformers/modeling_bert.py#L150

In the future, please make an effort to write a decent post that explains exactly what you need. This is very low quality.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or unsubscribe.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
