What is probability to get two balls of different colors, if one of them is blue? 
There are $12$ red, $8$ green and $10$ blue balls in a box. Two balls are taken out at random. What is probability to get two balls of different colors, if one of them is blue? 

We can use the formula of Conditional probability. Let $B=\{\text{got blue ball}\}$, $A = \{\text{two balls of different colors are taken}\}$.
$$P(A \mid B) = \frac{P(AB)}{P(B)}$$
$$P(B) = \frac{C^1_{10} \cdot C^1_{29}}{C^2_{30}}$$
$$P(AB)= \frac{C^1_{10} \cdot C^1_{20}}{C^2_{30}}$$
And answer is $20/29$. But it's wrong, and I don't understand why. Right is $40/49$.

You computed the probability of obtaining at least one blue ball incorrectly.  For the favorable cases, you can have either two blue balls, which occurs with probability 
$$\Pr(\text{two blue}) = \frac{\dbinom{10}{2}}{\dbinom{30}{2}}$$
or one blue ball and one ball of a different color, which you correctly calculated is 
$$\Pr(\text{exactly one blue}) = \frac{\dbinom{10}{1}\dbinom{20}{1}}{\dbinom{30}{2}}$$
so the probability that at least one of the balls is blue is 
$$\Pr(\text{at least one blue}) = \frac{\dbinom{10}{2} + \dbinom{10}{1}\dbinom{20}{1}}{\dbinom{30}{2}}$$
Thus, the probability that two different balls of different colors are obtained if at least one blue ball is selected is
\begin{align*}
\Pr(\text{two balls of different colors} \mid \text{at least one blue selected}) & = \frac{\frac{\binom{10}{1}\binom{20}{1}}{\binom{30}{2}}}{\frac{\binom{10}{2} + \binom{10}{1}\binom{20}{1}}{\binom{30}{2}}}\\
& = \frac{\dbinom{10}{1}\dbinom{20}{1}}{\dbinom{10}{2} + \dbinom{10}{1}\dbinom{20}{1}}
\end{align*}
In your attempt to calculate $\Pr(\text{at least one blue})$, you selected a blue ball and one of the remaining balls.  Doing so counts each selection with $2$ blue balls twice, once for each way you could designate one of the blue balls as the blue ball you have chosen.  Observe that 
$$\color{red}{\binom{2}{1}}\binom{10}{2} + \binom{10}{1}\binom{20}{1} = \color{red}{\binom{10}{1}\binom{29}{1}}$$
