about $\operatorname{SO}(2)$ group When 
$$
A =\begin{pmatrix} a & b \\ c & d \end{pmatrix},\quad  a,b,c,d \in \mathbb{R}, \quad A^2 -2aA + I = 0
$$
Is $ A \in \operatorname{SO}(2)\; $ if $A^n \in SO(2)\;$ for some $n \in \Bbb N$?
$\operatorname{SO}(2)$ is special orthogonal group.

The answer is yes I believe. Your requirement that $A^2-2aA+I=0$ forces $\det(A)=1$, so we know that if $A^n \in SO(2)$, then $\det(A^n)=\det(A)=1$, so we've eliminated the problem that Qiaochu mentioned. Now, it can be shown using a little bit of algebra that any matrix $B$ in $SO(2)$ is of the form $$\begin{pmatrix} \sin x & -\cos x \\ \cos x & \sin x \end{pmatrix}\ $$ Looking at a few powers of this matrix, we see that $$ B^2= \begin{pmatrix} -\cos 2x & -\sin 2x \\ \sin 2x & -\cos 2x \end{pmatrix}\ \text{and  } B^3= \begin{pmatrix} -\sin 3x & \cos 3x \\ -\cos 3x & -\sin 3x \end{pmatrix}\ $$ Thus, if we have $A^n \in SO(2)$, then if we finagle the signs and possibly switch $\sin $ and $\cos $ in a very natural way, it seems like $A$ will also be of the proper form with angle $\frac{x}{n}$, implying $A \in SO(2)$.
This isn't a rigorous proof, but I believe it at least gives the method of it.
Edit: As Shiyu pointed out, if $B$ is instead written in the form $$ \begin{pmatrix} \cos x & -\sin x \\ \sin x & \cos x \end{pmatrix}\ $$ then $$ B^k = \begin{pmatrix} \cos kx & -\sin kx \\ \sin kx & \cos kx \end{pmatrix}\ $$ so if $A^n \in SO(2)$ then $$ A^n = \begin{pmatrix} \cos y & -\sin y \\ \sin y & \cos y \end{pmatrix}\ \implies A= \begin{pmatrix} \cos \frac{y}{n} & -\sin \frac{y}{n} \\ \sin \frac{y}{n} & \cos \frac{y}{n} \end{pmatrix}\ $$ so $A \in SO(2)$
