Why are the embeddings of tokens multiplied by $\sqrt D$ (note not divided by square root of D) in a transformer? Why does the transformer tutorial in PyTorch have a multiplication by sqrt number of inputs? I know there is a division by sqrt(D) in the multiheaded self attention, but why is there something similar to with the output of the encoder? Especially because the original paper doesn't seem to mention it.
In particular (https://pytorch.org/tutorials/beginner/translation_transformer.html):
src = self.encoder(src) * math.sqrt(self.ninp)

or this (https://pytorch.org/tutorials/beginner/transformer_tutorial.html):
# helper Module to convert tensor of input indices into corresponding tensor of token embeddings
class TokenEmbedding(nn.Module):
    def __init__(self, vocab_size: int, emb_size):
        super(TokenEmbedding, self).__init__()
        self.embedding = nn.Embedding(vocab_size, emb_size)
        self.emb_size = emb_size

    def forward(self, tokens: Tensor):
        return self.embedding(tokens.long()) * math.sqrt(self.emb_size)

Note that I am aware that the Attention layer has this equation:
$$
\alpha = Attention(Q,K,V) = SoftMax( \frac{ Q K^\top }{\sqrt{D}} ) V
$$
and they argue why about it in the paper in a one of the margins (something about sum of variance being 1).
Is this related to that comment and how is it related? Is this mentioned in the original paper?
cross posted:

*

*https://discuss.pytorch.org/t/why-does-the-transformer-tutorial-have-a-multiplication-by-square-root-of-the-number-of-inputs/126738/6

*https://www.reddit.com/r/learnmachinelearning/comments/okfd7g/why_does_the_embedding_of_tokens_to_the/

*https://www.reddit.com/r/pytorch/comments/op0z2t/why_are_the_embeddings_of_tokens_multiplied_by/

*https://www.reddit.com/r/LanguageTechnology/comments/op13ep/why_are_the_embeddings_of_tokens_multiplied_by/

*https://www.reddit.com/r/deeplearning/comments/opcnwy/why_are_the_embeddings_of_tokens_multiplied_by/

*https://qr.ae/pGuVsC

We multiply because we use learned embeddings to convert the input tokens and output tokens to vectors of dimension d_model in the embedding layer. Refer Section 3.4 on Embeddings and Softmax of the original paper. You'd also see that an object of TokenEmbedding is used in Seq2SeqTransformer for source and target token embeddings.
