What is this relationship between trigonometric and hyperbolic function? In the following, I don't understand how they put $\tan{\phi} = \sinh{\frac{\psi}{\sqrt{2}}}$.

Is there a relationship?

This isn't asserting a relationship. It's simply setting up a substitution that trades circular functions for hyperbolics in order to escape the square root. It's really no different than having $\sqrt{4-x^2}$ and setting $x=2\sin\theta$.
In the exercise, first note that the expression (ignoring $d\theta$) can be rewritten as
$$a^2\left((1+\tan^2\phi)+\tan^2\phi\right)^{1/2} \sec^2\phi\, d\phi = a^2\left(1+2\tan^2\phi\right)^{1/2} \sec^2\phi\,d\phi$$
The author is suggesting that the form $1+2\tan^2\phi$ calls for the substitution
$$\tan\phi = \frac{\sinh\psi}{\sqrt{2}} \qquad\qquad \sec^2\phi\,d\phi=\frac{\cosh\psi}{\sqrt{2}}\,d\psi$$
(note that the entire $\sinh\psi$ is divided by $\sqrt{2}$, not just $\psi$) so that we have
$$\begin{align}
a^2\left(1+2\cdot\frac{\sinh^2\psi}{2}\right)^{1/2} \frac{\cosh\psi\,d\psi}{\sqrt{2}} &= \frac1{\sqrt{2}} a^2\left(1+\sinh^2\psi\right)^{1/2} \cosh\psi\,d\psi \\[4pt]
&= \frac1{\sqrt{2}}a^2\left(\cosh^2\psi\right)^{1/2} \cosh\psi\,d\psi \\[4pt]
&= \frac1{\sqrt{2}}a^2\cosh^2\psi\,d\psi \tag1
\end{align}$$
Alternatively, one could make a standard circular substitution thusly:
$$\tan\phi = \frac{\tan\omega}{\sqrt{2}} \qquad\qquad
\sec^2\phi\,d\phi = \frac{\sec^2\omega}{\sqrt{2}}d\omega$$
to give
$$\begin{align}
a^2\left(1+2\cdot\frac{\tan^2\omega}{2}\right)^{1/2} \frac{\sec^2\omega d\omega}{\sqrt{2}} &= \frac1{\sqrt{2}}a^2\left(1+\tan^2\omega\right)^{1/2} \sec^2\omega\,d\omega \\[4pt]
&= \frac1{\sqrt{2}}a^2\left(\sec^2\omega\right)^{1/2} \sec^2\omega\,d\omega \\[4pt] 
&= \frac1{\sqrt{2}}a^2\sec^3\omega\,d\omega \tag2
\end{align}$$
Expression $(1)$ has fairly straightforward integral, which in turn has a fairly straightforward back-substitution into $\phi$-form. Expression $(2)$ is perhaps a little messier on both counts. (The author may have anticipated this, or the author may have had some other reason to favor the hyperbolic substitution.) The end result in both cases is the same.
By the way, a "third" way to proceed is with this substitution:
$$\tan\phi = \frac{u}{\sqrt{2}} \qquad\qquad \sec^2\phi\,d\phi=\frac{du}{\sqrt{2}}$$
This gives
$$\begin{align}
a^2\left(1+2\cdot\frac{u^2}{2}\right)^{1/2} \frac{du}{\sqrt{2}} &= \frac1{\sqrt{2}}a^2\left(1+u^2\right)^{1/2}du \tag3
\end{align}$$
From here, substitution $u=\sinh\psi$ reverts to $(1)$, and $u=\tan\omega$ reverts to $(2)$, so this isn't necessarily a different approach. On the other hand, integration by parts and back-substituting are pretty straightforward from here, too.
