5. The solution set of the inequality
$$
\log _{\frac{1}{12}}\left(x^{2}+2 x-3\right)>x^{2}+2 x-16
$$

is
5. $(-5,-3) \cup(1,3)$.

Let $t=x^{2}+2 x-3$. Then the original inequality becomes
$$
\begin{array}{l}
\log _{\frac{1}{1}} t>t-13 . \\
\text { Let } f(t)=\log _{\frac{1}{1}} t-t .
\end{array}
$$

Thus, $f(t)$ is a decreasing function on the interval $(0,+\infty)$, and
$$
f(12)=-1-12=-13 \text {. }
$$

Therefore, the original inequality can be written as
$$
\begin{array}{l}
f(t)>f(12) \Leftrightarrow 0<t<12 \\
\Leftrightarrow 0<x^{2}+2 x-3<12 \\
\Leftrightarrow-5<x<-3 \text { or } 1<x<3 .
\end{array}
$$