Given ellipse of axes $a$ and $b$, find axes of tangential and concentric ellipse at angle $t$ Let’s say I have an ellipse with horizontal axis $a$ and vertical axis $b$, centered at $(0,0)$.
I want to compute $a’$ and $b’$ of a smaller ellipse centered at $(0,0)$, with the axes rotated by some angle $t$, tangent to the bigger ellipse and $\frac{a’}{b’}=\frac{a}{b}$.


Let the ratio $r= \frac{a’}a=\frac{b’}b$. Then, the inscribed ellipse with the tilt angle $t$ is
$$\frac{(x\cos t+y\sin t)^2}{r^2a^2}+ \frac{(-x\sin t+y\cos t)^2}{r^2b^2}=1\tag 1
$$
Also, the inscribed ellipse can be expressed as
$$
\frac{x^2}{a^2}+\frac{y^2}{b^2}-k\left(
  \frac{x\sin \theta}{a}-\frac{y\cos \theta}{b}\right)^2=1\tag 2$$
as pointed out by @Ng Chung Tak. Match the terms of $x^2$, $y^2$ and $xy$ between (1) and (2) to establish the system of equations below
\begin{align}k\sin 2\theta &=\frac{b^2-a^2}{r^2ab}\sin 2t\\
1-k\sin^2 \theta &= \frac1{r^2b^2}(b^2\cos^2 t+a^2\sin^2t)\\
 1-k\cos^2 \theta &= \frac1{r^2a^2}(a^2\cos^2 t+b^2\sin^2t)\\
\end{align}
Plug the 1st equation into the others to eliminate $k$
$$r^2- \frac{b^2\cos^2 t+a^2\sin^2t}{b^2}= \frac{b^2-a^2}{2ab}\sin 2t\tan\theta
$$
$$r^2- \frac{a^2\cos^2 t+b^2\sin^2t}{a^2}= \frac{b^2-a^2}{2ab}\sin 2t\cot\theta
$$
Then, multiply the two equations to eliminate $\theta$. After simplification
$$r^4 -( 2+\Delta^2)r^2+1=0$$
where $\Delta = \left( \frac a b - \frac b a\right)\sin t$. Solve to obtain the axis ratio
$$r= \sqrt{1+\frac{\Delta^2}4}-\frac{\Delta}2$$
Thus, the axes of the smaller ellipse are
$$a’ = ra, \>\>\>\>\>b’= rb$$
As expected, $r=1$ in the special case of $t=0$, and $r=\frac b a$ of $t=\frac\pi2$.
