We have occasionally seen many tricks which were used in various papers without any particular explanation.
For eg: in Attention is all you need you see the embedding multiplied by sqrt(embed_dim)
Are there any articles or notes which talks about various techniques that people have tried and what is the motivation behind these adhoc tuning tricks?