Solve $\int(1+\frac{1}{x})^{2}\frac{dx}{x^2}$ This seems to be a straightforward integral: 
$$\int(1+\frac{1}{x})^{2}\frac{dx}{x^2}$$
By substitution substitution, $u = 1+\frac{1}{x}, du = -\frac{1}{x^2}$
$$-\int u^2= -\frac{u^3}{3}+C = -\frac{(1+\frac{1}{x})^3}{3}$$
Then I took the derivative:
$$\frac{d}{dx}-\frac{(1+\frac{1}{x})^3}{3} = -\frac{1}{3}\frac{d}{dx}(1+\frac{1}{x})^3$$
If $u = 1+ \frac{1}{x}$  then $$-\frac{1}{3}\frac{d}{du}u^3 = -u^2 = -(1+\frac{1}{x})^2$$
What went wrong with my substitution?

You assumed $u=1+\dfrac 1x$. So far so good. 
But you didn't replace $dx$ by $du$ correctly.
You must not forget that $du= -\dfrac 1{x^2} dx$
So, 
$$\frac{d(1+\dfrac 1x)^3}{dx} = \frac{d(u^3)}{du} \color{red}{\cdot \frac{du}{dx}}$$
Always remember that
$$\color{blue}{\frac{d}{dx} f(x)= \frac{d}{du} f(x) \cdot \frac {du}{dx}}$$
