Transform curve into canonical form and determine type I have the curve:
$$9x^2-6xy+y^2+6x-2y-3=0$$
I have to transform it into the nnormal form and determine its type.
Well, we all know that such a curves are given by equation:
$$Ax^2+2Bxy+Cy^2+2Dx+2Ey+F = 0$$
Which is purely the case.

$\textbf{Thoughts:}$
1.) I determined if given curve is central or not:
$$\Delta=\begin{vmatrix} A&B\\B&C \end{vmatrix} = \begin{vmatrix}9&-3\\-3&1 \end{vmatrix} = 9 -9=0$$
Therefore, the curve is not central.

2.) Now have to find an angle of transformation, by the formula:
$$\tan^2(\varphi)+\frac{a_{11}-a_{22}}{a_{12}}\tan(\varphi)-1=0$$
Where:
$$\begin{array}{|c|c|c|}\hline a_{11}&a_{12}&a_{22}\\ \hline 9 & -3 & 1 \\ \hline \end{array}$$
Therefore:
$$\tan^2(\varphi)-\frac{8}{3}\tan(\varphi)-1=0$$
or
$$3\tan^2(\varphi)-8\tan(\varphi)-3=0$$
which is square equation and substitute $t=\tan(\varphi)$ we get:
$$3t^2-8t-1=0$$
And roots are $t_1=3$ and $t_2=-\frac{1}{3}$ (which is strange for tangent, in my opinion).

3.) Find transformation, and appropriate $\sin x$ and $\cos x$
It is well known that there are transformations:
$$\begin{cases} x = x_1\cos(\varphi)-y_1\sin(\varphi) \\ y= x_1\sin(\varphi)+y_1\cos(\varphi) \end{cases} \tag{1}$$
So for $\cos (\varphi)$: $$\cos(\varphi)=\frac{1}{\sqrt{1+\tan^2(\varphi)}}$$
And for $\sin (\varphi)$: $$\sin(\varphi) = \frac{\tan(\varphi)}{\sqrt{1+\tan^2(\varphi)}}$$
As it does not matter which root to apply here I take $3$ and get:
$$\cos(\varphi) = \frac{1}{\sqrt{10}}$$
and
$$\sin(\varphi) = \frac{3}{\sqrt{10}}$$
or, if substitute results into $(1)$ we get:
$$\begin{cases}x = x_1\frac{1}{\sqrt{10}}-y_1\frac{3}{\sqrt{10}} \\ y = x_1\frac{3}{\sqrt{10}}+y_1\frac{1}{\sqrt{10}} \end{cases} \tag{2}$$

4.) System $(2)$ shows us new coordinates, which we should substitute into original equarion, and have:
$$9\left(x_1\frac{1}{\sqrt{10}}-y_1\frac{3}{\sqrt{10}}\right)^2-6\left(x_1\frac{1}{\sqrt{10}}-y_1\frac{3}{\sqrt{10}}\right)\cdot \left(x_1\frac{3}{\sqrt{10}}+y_1\frac{1}{\sqrt{10}} \right) + \left(x_1\frac{3}{\sqrt{10}}+y_1\frac{1}{\sqrt{10}}\right)^2 + 6\left( x_1\frac{1}{\sqrt{10}}-y_1\frac{3}{\sqrt{10}}\right) - 2\left(x_1\frac{3}{\sqrt{10}}+y_1\frac{1}{\sqrt{10}}\right) - 3 = 0 \tag{3}$$
And I am totally stuck here

Hint:
Use Gauß' algorithm to write this polynomial as a sum of squares:
\begin{align}
9x^2-6xy+y^2+6x-2y-3&=(9x^2-6xy+6x)+y^2-2y-3\\
&=\bigl[(3x-y+1)^2-y^2+2y-1\bigr]+y^2-2y-3\\
&=(3x-y+1)^2-4\\
&=(3x-y-1)(3x-y+3).
\end{align}
