Question: $x^2-(2+i)x+(-1+7i)=0$

I tried to solve it and I got stuck here:

$x=(2+i)±\sqrt{\frac{7-24i}{2}}$

Answer: 
Firstly, you correctly found the discriminant:
$$D = B^2 - 4AC = [-(2+i)]^2-4(-1+7i) = 7 - 24i.$$
Then:
$$x\_{1,2} = \frac{-B\pm \sqrt{D}}{2A}=\frac{2+i\pm\sqrt{7-24i}}{2}.\tag{1}$$
But:
$$\sqrt{7-24i}=\pm(4-3i).$$ Why?

Let $\sqrt{7-24i} = z\implies z^2 = 7-24i$.
If we let $z = a+bi \implies a^2-b^2 +2ab i = 7-24i $. Thus:
$$\left\{
\begin{array}{l}
a^2 - b^2 = 7\\
ab = -12
\end{array}
\right.
$$
Solving the above system in Reals, we get 2 pairs of solutions: $(a,b) = (4,-3)$ and $(a,b) = (-4,3)$. Both $z\_1 = 4-3i$ and $z\_2 = -4+3i$ satisfy the equation $z^2 = 7-24i$. No matter the choice we make for $\sqrt{27-4i}$ (either $4-3i$ or $-4+3i$), the solutions given by the quadratic formula will be the same, due to the "$\pm$ sign" in the numerator.

Apply this to $(1)$ and you will get the result.
HINT
----

Write $7 - 24\iota $ in the $r e^{\iota \theta}$ form, where $r = \sqrt{7^2 + 24^2}$ and $ \tan{\theta} = -\frac{24}{7} $. Note, $\theta$ will be in 4th quadrant.