Why doesn't this approach work for $\int \sec^4 x\,dx$? I been trying to integrate $\sec^4$ , without much luck. But I don't entirely understand why my result is invalid and would like some feedback if possible.
I'm attacking the issue in the following way
$$
\int (\sec^2{x})^2dx = \int (\tan^2+1)^2dx
$$
then, I put $u=\tan^2+1$ which means $x = \arctan(\sqrt{u-1})$, which allows me to do the following backwards substitution
$$
\int (\sec^2{x})^2dx = \int (\tan^2+1)^2dx = \int u^2 \frac{1}{2u\sqrt{u-1}} dx = \frac{1}{2}\int u (u-1)^{\frac{-1}{2}} dx 
$$
Now using integration by parts I get
$$\begin{align*}
\int u (u-1)^{\frac{-1}{2}} dx &= \frac{1}{2}(2u(u-1)^{\frac{1}{2}} - \int 2(u-1)^{\frac{1}{2}})\\\\
&= \frac{1}{2}(2u(u-1)^{\frac{1}{2}}-\frac{4}{3}(u-1)^{3/2}) \\\\
&= \frac{1}{2}(2(\tan^2{x}+1)(\tan^2{x})^{\frac{1}{2}}-\frac{4}{3}(\tan^2{x})^{3/2})
\end{align*}$$
This however, seems to be incorrect. How come?

Your result is almost valid.
Taking a different approach, using Integration by Parts from the start:
Note that $\sec^2x = \frac{d}{dx} (\tan x)$.
We can use integration by parts:
$u = \sec^2 x \implies\,du = 2\sec^2 x \tan x\,dx$
$dv = \sec^2 x \,dx\implies \, v = \tan x$.
$$\int \sec^4 x \,dx = \sec^2x\tan x - 2\int \tan^2 x \sec^2 x$$
Now, the remaining integral can be easily solved by substitution: $w = \tan x,\;\implies dw = \sec^2 x$: $$2\int \tan^2 x \sec^2 x \,dx = 2\int w^2 \,dw = \dfrac 23 w + c$$

Putting the above together gives us:
$$\int \sec^4 x \,dx = \sec^2x\tan x - \frac 23 \tan^3 x + C$$

Note: Your answer is very, very close, and can be manipulated algebraically to closely match the above:
$$\frac{1}{2}\Big(2(\underbrace{\tan^2{x}+1}_{\sec^2 x})\underbrace{(\tan^2{x})^{\frac{1}{2}}}_{|\tan x|}-\frac{4}{3}\underbrace{(\tan^2{x})^{3/2}}_{|\tan^3 x|}\Big)
= \sec^2 |\tan x| - \frac 23 |\tan^3 x| + C$$
