Inverse of Roto-Translation Matrix I'm trying to implement two JavaScript functions to calculate


*

*A 2D coordinate transformation from the reference frame $O$ to $O'$;

*The inverse transformation from $O'$ to $O$;


The first step it's very simple for me and it works. My mathematical process is this:
$$
  \begin{pmatrix}
    x'\\
    y'\\
    1
  \end{pmatrix}
=
  \begin{pmatrix}
    R & u_0\\
    0 & 1
  \end{pmatrix}
  \begin{pmatrix}
    u\\
    0
  \end{pmatrix}
=
  \begin{pmatrix}
    cos\theta & -sin\theta & x_0\\
    sin\theta & cos\theta & y_0\\
    0 & 0 & 1
  \end{pmatrix}
  \begin{pmatrix}
    x\\
    y\\
    1
  \end{pmatrix}
$$
Where $R$ is the rotation matrix, $u_0$ is the $O'$ origin in the $O$ frame reference and $u$ is a point in $O$. At the end I get this two equations:
$$
\left\{ 
\begin{array}{c}
    x' = xcos\theta - ysin\theta + x_0\\
    y' = xsin\theta + ycos\theta + y_0\\
\end{array}
\right. 
$$
The problem is how to calculate the inverse transformation. I'm doing something like this but it doesn't work:
$$
  \begin{pmatrix}
    x\\
    y\\
    1
  \end{pmatrix}
=
  \begin{pmatrix}
    R^{-1} & -R^{-1}u_0\\
    0 & 1
  \end{pmatrix}
  \begin{pmatrix}
    u'\\
    0
  \end{pmatrix}
=
  \begin{pmatrix}
    \frac{1}{cos\theta} & sin\theta & -\frac{x_0}{cos\theta}-y_0sin\theta\\
    -sin\theta & \frac{1}{cos\theta} & {x_0}{sin\theta}-\frac{y_0}{cos\theta}\\
    0 & 0 & 1
  \end{pmatrix}
  \begin{pmatrix}
    x'\\
    y'\\
    1
  \end{pmatrix}
$$
The final equations are:
$$
\left\{ 
\begin{array}{c}
    x = x'\frac{1}{cos\theta} + y'sin\theta -\frac{x_0}{cos\theta}-y_0cos\theta\\
    y = -x'sin\theta + y'\frac{1}{cos\theta} + {x_0}{sin\theta}-\frac{y_0}{cos\theta}\\
\end{array}
\right. 
$$
Can you tell me where I wrong?

Pay attention to what transformation you want to represent, whether active or passive. 
Anyway, if $R$ is the matrix that represents a rotation of angle $\theta$ in a plane, its inverse is the matrix that represents a rotation of the opposite angle $- \theta$, around the same origin, and it is easy to see that it is represented by the transpose matrix: $R^{-1}=R^T$ (it seems that this is your mistake, when you use $\frac{1}{\cos \theta}$ in the inverse matrix.). 
From this it is also easy to see that the inverse  of a rigid tranformation $T$ that , in homogeneous coordinates, is represented by the matrix 
$$
T=\begin{pmatrix}
R&\vec u\\
0&1
\end{pmatrix}
$$
is represented by the matrix
$$
T^{-1}=\begin{pmatrix}
R^T&-R^T\vec u\\
0&1
\end{pmatrix}
$$
