I created one post for a similar question here,

  
    
    
    How to add padding mask to nn.TransformerEncoder module? nlp
  
  
    I want to use vanilla transformer(only the encoder side), but I don’t know how&where to add the padding mask.
  


see if it helps, regarding math.sqrt(self.ninp)
in the paper https://arxiv.org/pdf/1706.03762.pdf 16, they use a scaling factor of math.sqrt(self.ninp), maybe cancelling this scaling factor (or not considering to divide by math.sqrt(self.ninp)), gives a better accuracy in the tutorial.