Hi, I’d like to add that I’ve also tried the built in PyTorch TransformerEncoder class for a classification task, it seems to learn nothing, whereas this one 30 by lucidrains works fine. Is there a bug in the native implementation? Or perhaps some missing features that the others may have?