Question: $x^2-(2+i)x+(-1+7i)=0$

I tried to solve it and I got stuck here:

$x=(2+i)±\sqrt{\frac{7-24i}{2}}$

Answer: 
Firstly, you correctly found the discriminant:
$$D = B^2 - 4AC = [-(2+i)]^2-4(-1+7i) = 7 - 24i.$$
Then:
$$x\_{1,2} = \frac{-B\pm \sqrt{D}}{2A}=\frac{2+i\pm\sqrt{7-24i}}{2}.\tag{1}$$
But:
$$\sqrt{7-24i}=\pm(4-3i).$$ Why?

Let $\sqrt{7-24i} = z\implies z^2 = 7-24i$.
If we let $z = a+bi \implies a^2-b^2 +2ab i = 7-24i $. Thus:
$$\left\{
\begin{array}{l}
a^2 - b^2 = 7\\
ab = -12
\end{array}
\right.
$$
Solving the above system in Reals, we get 2 pairs of solutions: $(a,b) = (4,-3)$ and $(a,b) = (-4,3)$. Both $z\_1 = 4-3i$ and $z\_2 = -4+3i$ satisfy the equation $z^2 = 7-24i$. No matter the choice we make for $\sqrt{27-4i}$ (either $4-3i$ or $-4+3i$), the solutions given by the quadratic formula will be the same, due to the "$\pm$ sign" in the numerator.

Apply this to $(1)$ and you will get the result.
> 
> By using factor method:
> 
> 
> 

$$x^2-(2+i)x+(-1+7i)=0$$

* Rewrite the left hand side of the equation.

$$x^2-(2+i)x+(-1+7i)=(-1+7i)+(-2-i)x+x^{2}$$

* Factor the left hand side.

$$(( -3+i)+x)((1-2i)+x)=0$$
 - Solve each term in the product separately.

$(-3+i)+x=0$ or $(1-2i)+x=0$

$$x=3-i \text{ or } x=-1+2i $$

> 
> Or we can Solve the quadratic equation by completing the square
> 
> 
> 

indeed,

$$x^2-(2+i)x+(-1+7i)=0$$

Subtract $-1+7i$ from both sides:

$$x^2-(2+i)x=1-7i$$
Take one half of the coefficient of $x$ and square it, then add it to both sides

Add $\dfrac{3}{4}+i$ to both sides:

$$(\dfrac{3}{4}+i)+(-2-i)x+x^{2}=\dfrac{7}{4}-6i$$

* Factor the left hand side.

Write the left hand side as a square:

$$\left(x+(-1-\dfrac{i}{2})\right)^{2}=\dfrac{7}{4}-6i$$

* Eliminate the exponent on the left hand side.

Take the square root of both sides:

$$x+(-1-\dfrac{i}{2})=\sqrt{\dfrac{7}{4}-6i} \text{ Or } x+(-1-\dfrac{i}{2})=-\sqrt{\dfrac{7}{4}-6i}$$

add $1+\dfrac{i}{2}$ to both sides: 

$$x=(1+\dfrac{i}{2})+\sqrt{\dfrac{7}{4}-6i} \text{ Or } x=(1+\dfrac{i}{2})-\sqrt{\dfrac{7}{4}-6i}$$

$$x=(1+\dfrac{i}{2})+\sqrt{4-6i-\dfrac{9}{4}} \text{ Or } x=(1+\dfrac{i}{2})-\sqrt{4-6i-\dfrac{9}{4}}$$

$$x=(1+\dfrac{i}{2})+\sqrt{\dfrac{(4-3i)^{2}}{4}} \text{ Or } x=(1+\dfrac{i}{2})-\sqrt{\dfrac{(4-3i)^{2}}{4}}$$

$$x=(1+\dfrac{i}{2})+\dfrac{(4-3i)}{2} \text{ Or } x=(1+\dfrac{i}{2})-\dfrac{(4-3i)}{2}$$