Question: $x^2-(2+i)x+(-1+7i)=0$

I tried to solve it and I got stuck here:

$x=(2+i)±\sqrt{\frac{7-24i}{2}}$

Answer: 
Firstly, you correctly found the discriminant:
$$D = B^2 - 4AC = [-(2+i)]^2-4(-1+7i) = 7 - 24i.$$
Then:
$$x\_{1,2} = \frac{-B\pm \sqrt{D}}{2A}=\frac{2+i\pm\sqrt{7-24i}}{2}.\tag{1}$$
But:
$$\sqrt{7-24i}=\pm(4-3i).$$ Why?

Let $\sqrt{7-24i} = z\implies z^2 = 7-24i$.
If we let $z = a+bi \implies a^2-b^2 +2ab i = 7-24i $. Thus:
$$\left\{
\begin{array}{l}
a^2 - b^2 = 7\\
ab = -12
\end{array}
\right.
$$
Solving the above system in Reals, we get 2 pairs of solutions: $(a,b) = (4,-3)$ and $(a,b) = (-4,3)$. Both $z\_1 = 4-3i$ and $z\_2 = -4+3i$ satisfy the equation $z^2 = 7-24i$. No matter the choice we make for $\sqrt{27-4i}$ (either $4-3i$ or $-4+3i$), the solutions given by the quadratic formula will be the same, due to the "$\pm$ sign" in the numerator.

Apply this to $(1)$ and you will get the result.
Some other answers explained how to get square roots of complex numbers, but it's also worth mentioning that there is an explicit formula you can get by following such methods in generality (leaving the variables in). The formula obtained is that the square root of $a + bi$ with nonnegative real part is given by

$$\sqrt{a + bi} = \sqrt{{a + \sqrt{a^2 + b^2} \over 2}} \pm \sqrt{{-a + \sqrt{a^2 + b^2} \over 2}}\,\,i$$

You choose the $\pm$ to ensure your square root is in the correct quadrant. So in your case

$$\sqrt{7 - 24i} = \sqrt{{7 + \sqrt{625} \over 2}} \pm \sqrt{{-7 + \sqrt{625} \over 2}}\,i$$
$$= 4 \pm 3i$$
Based on $7 - 24i$'s position in the fourth quadrant, you'd choose the $-$ in the $\pm$, so you'd have
$$\sqrt{7 - 24i} = 4 - 3i$$
This is the square root with positive real part. $-4 + 3i$ is the square root with negative real part; you just multiply the other square root by $-1$. Due to the $\pm$ in the quadratic formula, it doesn't matter which square root you use when solving quadratic equations.