@david.waterworth thanks for posting! Woah, I believe the Positional Encoding module is a must since it actually gives order to the words you input into the model. If you included a sentence without Positional Encoding, that basically means you’re giving the model a bag of words instead of a sequence of words. Regardless, it might be that there is a bug in the Positional Encoding - maybe. Could you please post your code in a google colab codebook or simply post your model here?