Finding $P$ such that $P^TAP$ is a diagonal matrix 
Let $$A = \left(\begin{array}{cc} 2&3 \\ 3&4  \end{array}\right) \in
M_2(\mathbb{C})$$
Find $P$ such that $P^TAP = D$ where $D$ is a diagonal matrix.

So here's the solution:
$$A = \left(\begin{array}{cc|cc} 2&3&1&0\\ 3&4&0&1    \end{array}\right) \sim \left(\begin{array}{cc|cc} 2&0&1&-3/2\\ 0&-1/2&0&1    \end{array}\right)$$
Therefore, $$P = \left(\begin{array}{cc} 1&-3/2\\ 0&1    \end{array}\right) \\ P^TAP = \left(\begin{array}{cc} 2&0\\ 0&-1/2    \end{array}\right) $$
What was done here exactly? I'd be glad elaborate about the process.
Thanks.

Hermite Reduction.
SEE ALSO Orthogonal basis for this indefinite symmetric bilinear form
Transforming quadratic forms, how is this theorem called?
What is the difference between using $PAP^{-1}$ and $PAP^{T}$ to diagonalize a matrix?
When you have a symmetric matrix of integers, you may use Hermite's method for diagonalizing, the order they want is $P^t A P = D.$ Alright, I will need to do an inverse at the end.
Make a column vector 
$$
V =
\left(
\begin{array}{c}
x \\
y
\end{array}
\right)
$$ 
and write out
$$  V^T A V = 2 x^2 + 6 xy + 4 y^2  $$
Next, we cancel out all $x$ terms using 
$$ \left( x + \frac{3}{2} y \right)^2 = x^2 + 3 xy  + \frac{9}{4} y^2, $$ and
$$ 2 \left( x + \frac{3}{2} y \right)^2 = 2x^2 + 6 xy  + \frac{9}{2} y^2. $$
As a result,
$$ 2 \left( x + \frac{3}{2} y \right)^2 -   \frac{1}{2} y^2 =  2 x^2 + 6 xy + 4 y^2 . $$
MORE TYPING TO COME !!!!
In matrices, the direction I did is
$$
\left(
\begin{array}{cc}
1 & 0 \\
 \frac{3}{2} & 1
\end{array}
\right)
\left(
\begin{array}{cc}
2 & 0 \\
0 & -\frac{1}{2}
\end{array}
\right)
\left(
\begin{array}{cc}
1 & \frac{3}{2} \\
0 & 1
\end{array}
\right) =
\left(
\begin{array}{cc}
2 & 3 \\
3 & 4
\end{array}
\right)
$$
With
$$ Q =
\left(
\begin{array}{cc}
1 & \frac{3}{2} \\
0 & 1
\end{array}
\right)
$$
notice that the rows correspond exactly to the linear substitutions, the first row means $x + \frac{3}{2} y$ and the second row means $y.$
EVEN MORE EXCITING TYPING ANY MINUTE !!!!!!!!!!
What I did so far is in the order $Q^T D Q = A.$ All we need to do is take $p= Q^{-1},$ which is easier than usual because $\det Q = 1.$ The result is
$$
\left(
\begin{array}{cc}
1 & 0 \\
 -\frac{3}{2} & 1
\end{array}
\right)
\left(
\begin{array}{cc}
2 & 3 \\
3 & 4
\end{array}
\right)
\left(
\begin{array}{cc}
1 & -\frac{3}{2} \\
0 & 1
\end{array}
\right) =
\left(
\begin{array}{cc}
2 & 0 \\
0 &  -\frac{1}{2}
\end{array}
\right)
$$
The second example in the question, with 3 by 3 matrix, is
$$ x^2 + 4 y^2 + 4 z^2 + 16 yz + 4 zx + 4 xy.  $$ This is an example where an extra trick must be used:
$$ (x+2y+2z)^2 =  x^2 + 4 y^2 + 4 z^2 + 8 yz + 4 zx + 4 xy.   $$
All that remains to construct is $8yz$ because we used up the $y^2$ and $z^2.$ The trick is that $(y+z)^2 - (y-z)^2 = 4yz,$ so
$$ (x+2y+2z)^2 + 2 (y+z)^2 -2 (y-z)^2=  x^2 + 4 y^2 + 4 z^2 + 16 yz + 4 zx + 4 xy.   $$
Thus the diagonal matrix gets entries $1,2,-2$ and, in this direction,
$$
Q = 
\left( 
\begin{array}{ccc}
1 & 2 & 2 \\
0 & 1 & 1 \\
0 & 1 & -1
\end{array}
\right)
$$
and then $P = Q^{-1}$
$$
P = 
\left( 
\begin{array}{ccc}
1 & -2 & 0 \\
0 & \frac{1}{2} &  \frac{1}{2} \\
0 &  \frac{1}{2} & - \frac{1}{2}
\end{array}
\right)
$$
