Isolate yaw-pitch-roll from rotation I have a transformation that acts as such:
$$RX=Y$$
Where $R_{3\times3}$ is a yaw-pitch-roll rotation matrix, and $X, Y$ are 3D vectors.
Yaw ($\alpha$)-pitch ($\beta$)-roll ($\gamma$) rotations are the product of the 3 rotation matrices:
$$R_x(\gamma)R_y(\beta)R_z(\alpha)$$
Given $X$ and $Y$, how can I find $\alpha$, $\beta$ and $\gamma$?

The first answer to any question on how to find Euler angles is: don't; use quaternions instead.
If external factors force you to use Euler angles, you could proceed like this: If $X$ and $Y$ are identical, the rotation can be a rotation around $X$ through any angle. Otherwise, the rotation can be a rotation through any axis $A$ in the plane $A\cdot X=A\cdot Y$. If $X$ and $Y$ are antipodal, the rotation must be through an angle $\pi$. Otherwise, that plane is spanned by the two vectors $X+Y$ and $X\times Y$, so you can parametrize the possible axes as 
$$A(\xi)=\cos\xi\frac{X+Y}{|X+Y|}+\sin\xi\frac{X\times Y}{|X\times Y|}$$ with $\xi\in[0,\pi[$. The rotation angle must be the angle between the perpendicular projections of $X$ and $Y$ into the plane perpendicular to the axis, which are
$$
\begin{eqnarray}
X_\perp&=&
X - (A\cdot X)A\\
&=&X-\cos\xi\frac{X^2+X\cdot Y}{|X+Y|}A\\
&=&X-\frac12\cos\xi\frac{(X+Y)^2}{|X+Y|}A\\
&=&X-\frac12\cos\xi|X+Y|A
\end{eqnarray}$$
(where I used $X^2=Y^2$), and likewise for $Y_\perp$, so the angle between them is
$$
\begin{eqnarray}
\phi(\xi)
&=&\arccos\frac{X_\perp\cdot Y_\perp}{|X_\perp||Y_\perp|}\\
&=&\arccos\frac{X_\perp\cdot Y_\perp}{X_\perp\cdot X_\perp}\\
&=&\arccos\frac{X\cdot Y -\frac12\cos\xi|X+Y|A\cdot(X+Y)+\frac14\cos^2\xi |X+Y|^2}{X\cdot X -\frac12\cos\xi|X+Y|A\cdot(X+X)+\frac14\cos^2\xi |X+Y|^2}\\
&=&\arccos\frac{X\cdot Y -\frac14\cos^2\xi (X+Y)^2}{X\cdot X -\frac14\cos^2\xi (X+Y)^2}\\
&=&\arccos\frac{\cos^2\xi (X+Y)^2-4X\cdot Y}{\cos^2\xi (X+Y)^2-4X\cdot X}\\
&=&\arccos\frac{\sin^2\theta-\cos^2\theta\sin^2\xi}{\cos^2\theta\cos^2\xi-1}\;,\\
\end{eqnarray}
$$
where $\theta$ is the angle between $X$ or $Y$ and $X+Y$, that is, half the angle between $X$ and $Y$.
Using $A(\xi)$ and $\phi(\xi)$, you can construct the rotation matrices $R(\xi)$ and extract the Euler angles.
