Type of this Conic section I want to determine, to which type the following Conic sections belong to:
$$
\begin{align}
\textrm{(i)}&\quad-8x^2+12xy-6x+8y^2-18y+8=0\\
\textrm{(ii)}&\quad5x^2-8xy+2x+5y^2+2y+1=0
\end{align}
$$
To (i)
Matrix notation:
$$
\begin{pmatrix}x&y\end{pmatrix}\begin{pmatrix}-8&6\\6&8\end{pmatrix}\begin{pmatrix}x\\y\end{pmatrix}+2\begin{pmatrix}-3&-9\end{pmatrix}\begin{pmatrix}x\\y\end{pmatrix}+8=0\\
$$
$A=\begin{pmatrix}-8&6\\6&8\end{pmatrix}$
Eigenvalues are $\lambda_1=10,\lambda_2=-10$
Eigenvectors: $v_{\lambda_1}=\begin{pmatrix}1\\3\end{pmatrix}, v_{\lambda_2}=\begin{pmatrix}-3\\1\end{pmatrix}$
diagonal matrix $D=\begin{pmatrix}10&0\\0&-10\end{pmatrix}$
rotation matrix $B=\begin{pmatrix}\frac{1}{\sqrt{10}}&\frac{-3}{\sqrt{10}}\\\frac{3}{\sqrt{10}}&\frac{1}{\sqrt{10}}\end{pmatrix}$
Let $x=Bx'$
$\Rightarrow x^TAx+u^Tx=x'^TB^TABx'+u^TBx'=x'^TDx'+u^TBx'$
Calculating $u'^T=u^TB$
$u'^T=2(\begin{pmatrix}-3&-9\end{pmatrix}\begin{pmatrix}\frac{1}{\sqrt{10}}&\frac{-3}{\sqrt{10}}\\\frac{3}{\sqrt{10}}&\frac{1}{\sqrt{10})}\end{pmatrix}
)=\begin{pmatrix}-6\sqrt{10}&0\end{pmatrix}$
So the transformed quadric is:
$10x'^2-6\sqrt{10}x'-10y'^2+8=0$
I know, that the solution must be $10x^2-10y^2=1$, but completing the square didn't get  me there... Is my work so far correct? Where is my mistake? Can somebody please help me?

Well the second equation
$$ 5 x^2 - 8 x y + 5 y^2 + 2 x + 2 y + 1 = 0 $$
can be written as
$$
5 \Big( \big[ x - x_o \big] + x_o \Big)^2
- 8 \Big( \big[ x - x_o \big] + x_o \Big) \Big( \big[ y - y_o \big]
+ y_o \Big) + 5 \Big( \big[ y - y_o \big] + y_o \Big)^2\\
+ 2 \Big( \big[ x - x_o \big] + x_o \Big)
+ 2 \Big( \big[ y - y_o \big] + y_o \Big) + 1 = 0
$$
so
$$
5 \big[ x - x_o \big]^2
- 8 \big[ x - x_o \big] \big[ y - y_o \big]
+ 5 \big[ y - y_o \big]\\
+ \Big( 10 x_o - 8 y_o + 2 \Big) \big[ x - x_o \big]
+ \Big( 10 y_o - 8 x_o + 2 \Big) \big[ y - y_o \big]\\
+ 5 x_o^2 - 8 x_o y_o + 5 y_o^2 + 2 x_o + 2 y_o + 1 = 0
$$
So $x_o = -1$ and $y_o = -1$ gives
$$
5 \big[ x + 1 \big]^2
- 8 \big[ x + 1 \big] \big[ y + 1 \big]
+ 5 \big[ y + 1 \big]^2 = 1\\
$$
This can be written as
$$
9 \big[ x + 1 \big]^2
- 18 \big[ x + 1 \big] \big[ y + 1 \big]
+ 9 \big[ y + 1 \big]^2\\
+ \big[ x + 1 \big]^2
+ 2 \big[ x + 1 \big] \big[ y + 1 \big]
+ \big[ y + 1 \big]^2 = 2\\
$$
so
$$
9 \big( x - y \big)^2 + \big( x + y + 2 \big)^2 = 2
$$
which is an ellipse with the axis $x-y=0$ and $x+y+2=0$.
