What is a "symmetrical product" of an irreducible representation with itself? In their classic paper outlining the Jahn–Teller theorem (Proc. R. Soc. Lond. A 1937, 161 (905), 220–235), the authors wrote "where $[\phi^2]$ denotes the representation of the symmetrical product of $\phi$ with itself". Reading this I'm thinking about a regular product of representation (scalar product with each characters).

But in the next paragraph there is this equality for the $D_\mathrm{4h}$ group:

$$[\mathrm{E_g^2}] = [\mathrm{E_u^2}] = \mathrm{A_{1g} + B_{1g} + B_{2g}}$$

Obviously something is wrong, as the direct product is a 4-dimensional representation, but the right-hand side is 3-dimensional.

I would like to add that there is a reference of Weyl 1928 unfortunately this article is in German and I cannot find anything in it.

What is the correct way to carry out this direct product?
In your case you need the equations for the characters as there are symmetric and antisymmetric parts in the product. Your question gives just the symmetric part and the antisymmetric part is $A_2$ normally written as $[A_2]$ to indicate this. 

The characters are found using $\chi^+ =\left([\chi(R)]^2+ \chi(R^2) \right)/2$ for the symmetrical part and $\chi^- =\left([\chi(R)]^2- \chi(R^2) \right)/2$ for the antisymmetric part. Using the point group table you need to work out the square of all the operations for $E_g$ this gives $\chi(R^2)$ (e.g. $(C_2)^2 \to E$) and then square the operations which gives $[\chi(R)]^2$, i.e. $[E]^2\to 4$. From the list of characters you then identify the symmetry species, either directly by inspection (as for the antisymmetric in this case), or by forming the irreps from the reducible representation.