Question: Suppose that $x^\*$ means $5x^2 - x$. Then what does $(y+3)^\*$ mean?

Answer: 
If $x^\* = 5 x^2 - x$ then

$$(y+3)^\* = 5(y+3)^2 - (y+3 )= 5(y^2 + 6 y + 9) - y - 3 = 5 y^2 + 29 y + 42 $$
By 5x2 do you mean $5x^2$? In such a case, simply replace every instance of $x$ with $(y+3)$, and then expand.

For example, $f(x) = 2x+2 \Longrightarrow f(y+3) = 2(y+3)+2 = 2y+6+2 = 2y+8$.