$\int_{0}^{x^2 (1+x)} f(t) dt = x$ According to the second fundamental theorem of Calculus, for this problem $F(x^2(1+x))-F(0)=x$ can be used.
But I can't find a function F that works for this problem. Can someone help me with this?

Hint. You already have $$F(x^2(1+x))-F(0)=x.$$ Now, differentiate both sides to get $$x(3x+2)f(x^2(1+x))=1.$$
You can now solve for $f(x^2(1+x))$ explicitly, from which you can solve for $f(x)$, as long as you choose your domain wisely (since $x\mapsto x^2(1+x)$ is not a bijection from $\mathbb R$ to itself).

To do the last part, let's substitute $y=x^2(1+x)$, so that $x^3+x^2-y=0$. We want to solve this for $x$. This is quite troublesome to solve analytically (since it's a cubic), but can be done: let $x=1/a$ and multiply throughout by $a^3$ to get
$$1+a-ya^3=0.$$
The idea here is to use the triple angle identity for cosine, $\cos(3\theta)=4\cos^3\theta-3\cos\theta$. We substitute $a=k\cos\theta$ for some constant $k$ to get
$$yk^3\cos^3\theta-k\cos\theta+1=0.$$
We need the ratio of coefficients of $\cos^3\theta$ and $\cos\theta$ to be $4:-3$, so that $yk^2=4/3$, or $k=\sqrt{4/(3y)}$. So
$$\frac43\sqrt{\frac{4}{3y}}\cos^3\theta-\sqrt{\frac{4}{3y}}\cos\theta+1=0\iff\frac{2}{3\sqrt{3y}}\cos(3\theta)=-1.$$
Finally we can solve for $\cos(3\theta)$ in terms of $y$. Then, once we appropriately restrict the domain, we can find $\theta$ and hence $a$ and hence $x$ analytically. Once we have $x$ in terms of $y$, substituting back into
$$f(x^2(1+x))=\frac1{x(3x+2)}$$
will give us the answer, since the LHS will become $f(y)$ and the RHS will become something in terms of $y$. This provides an explicit formula for $f$.
