Just curious: Did the previous implementations of Bert from HuggingFace used any sin/cos in the positional embedding? It sounds like the vanilla Embedding layer was added later since it looks like it has more pros in replace of sin/cos positional embeddings.