Anandh_Perumal_Konar:

Regarding the scaling factor, even if we don’t scale it shouldn’t affect the results a lot, I can’t recollect the paper but in the earlier version of dot product attention even they didn’t use the scale factor. But obviously scaling helps.
But I don’t understand why are we multiplying it? it should be divided based on the attention paper.


Again, regarding the scaling factor:
The paper 2 does say:

In the embedding layers, we multiply those weights by sqrt(d_model).

(Note The Annotated Transformer 2 directly connects the paper to PyTorch code line-by-line).
However, the paper does not say why this is done.
This post 8 says the following:

The reason we increase the embedding values [by adding math.sqrt(self.d_model)] before addition is to make the positional encoding relatively smaller. This means the original meaning in the embedding vector won’t be lost when we add them together.