Hello!
I can’t figure out why the positional embeddings are implemented as just the vanilla Embedding layer in both PyTorch 26 and Tensorflow 25. Based on my current understanding, positional embeddings should be implemented as non-trainable sin/cos or axial positional encodings (from reformer).
Can anyone please enlighten me with this? Thank you so much!