Why are the embeddings of tokens multiplied by $\sqrt D$ (note not divided by square root of D) in a transformer? Why does the transformer tutorial in PyTorch have a multiplication by sqrt number of inputs? I know there is a division by sqrt(D) in the multiheaded self attention, but why is there something similar to with the output of the encoder? Especially because the original paper doesn't seem to mention it.
In particular (https://pytorch.org/tutorials/beginner/translation_transformer.html):
src = self.encoder(src) * math.sqrt(self.ninp)

or this (https://pytorch.org/tutorials/beginner/transformer_tutorial.html):
# helper Module to convert tensor of input indices into corresponding tensor of token embeddings
class TokenEmbedding(nn.Module):
    def __init__(self, vocab_size: int, emb_size):
        super(TokenEmbedding, self).__init__()
        self.embedding = nn.Embedding(vocab_size, emb_size)
        self.emb_size = emb_size

    def forward(self, tokens: Tensor):
        return self.embedding(tokens.long()) * math.sqrt(self.emb_size)

Note that I am aware that the Attention layer has this equation:
$$
\alpha = Attention(Q,K,V) = SoftMax( \frac{ Q K^\top }{\sqrt{D}} ) V
$$
and they argue why about it in the paper in a one of the margins (something about sum of variance being 1).
Is this related to that comment and how is it related? Is this mentioned in the original paper?
cross posted:

*

*https://discuss.pytorch.org/t/why-does-the-transformer-tutorial-have-a-multiplication-by-square-root-of-the-number-of-inputs/126738/6

*https://www.reddit.com/r/learnmachinelearning/comments/okfd7g/why_does_the_embedding_of_tokens_to_the/

*https://www.reddit.com/r/pytorch/comments/op0z2t/why_are_the_embeddings_of_tokens_multiplied_by/

*https://www.reddit.com/r/LanguageTechnology/comments/op13ep/why_are_the_embeddings_of_tokens_multiplied_by/

*https://www.reddit.com/r/deeplearning/comments/opcnwy/why_are_the_embeddings_of_tokens_multiplied_by/

*https://qr.ae/pGuVsC

Note that the weights in word embeddings get initialized with zero mean and unit variance. Also, the embeddings get multiplied by sqrt(d) before being added to the positional encoding. The positional encoding is also on the same scale as the embeddings.
My hypothesis is that the authors tried rescaling the embeddings by various numbers (as they certainly did with attention), and this particular rescaling happened to work, because it made the embeddings much bigger than the positional encodings (initially). The positional encodings are necessary, but they probably shouldn't be as "loud" as the words themselves.
