I’m not an expert in NLP (so double check with an expert), but I believe the square root is to normalize the encoding with respect to the input size so the embedding isn’t dependent on the size of the input. As you’re taking a dot product, the magnitude of the dot product with scale with the square-root of the dimension size and by dividing by that value you normalize it!