vainaijr:

not considering to divide by math.sqrt(self.ninp)


Well, actually my question is different from the one you asked.
My question is what’s the need for mask in the encoder layer?
But I think it makes sense when you want to use an architecture like GPT 2, I guess. Correct me if I’m wrong.
Regarding the scaling factor, even if we don’t scale it shouldn’t affect the results a lot, I can’t recollect the paper but in the earlier version of dot product attention even they didn’t use the scale factor. But obviously scaling helps.
But I don’t understand why are we multiplying it? it should be divided based on the attention paper.