Solved by utkuumetin in post #2 
              
              
                Hi, i’m not expert about pytorch or transformers but i think nn.Transformer doesn’t have positional encoding, you have to code yourself then to add token embeddings.