Question: I'm trying to solve:

$\hspace{40mm}$ $\lim\_{n\to\infty}$ $(\frac{\sqrt{n^4-2n^3} - n^2}{10n+2})^2$

This is what I did to try to solve it, but my answer doesn't match the possible answers so I know it's wrong.

$\lim\_{n\to\infty}$ $(\frac{\sqrt{n^4-2n^3} - n^2}{10n+2})^2$

* Highest power of denominator is n, so pull out $n^2$ from the square root to give an n

$\lim\_{n\to\infty}$ $(\frac{n(\sqrt{n^2-2n} )- n^2}{10n+2})^2$

* Factor out n from numerator and denominator

$\lim\_{n\to\infty}$ $(\frac{n((\sqrt{n^2-2n})- n)}{n(10+\frac{2}{n})})^2$

* Cancel n from numerator and denominator

$\lim\_{n\to\infty}$ $(\frac{\sqrt{n^2-2n} - n}{10+\frac{2}{n}})^2$

* Take $n^2$ out from the square root

$\lim\_{n\to\infty}$ $(\frac{n\sqrt{1-\frac{2}{n}} - n}{10+\frac{2}{n}})^2$

* Factor out n from numerator

$\lim\_{n\to\infty}$ $(\frac{n((\sqrt{1-\frac{2}{n}}) - 1)}{10+\frac{2}{n}})^2$

At this point I would take the limit to be 0, but the possible answers are 0.05, 0.01, 0.001, or 0.1. Where did I go wrong?

Answer: 
You have got everything right till the end. Use the fact that $(1-\frac 2 n)^{1/2} =1-\frac 1 n +o(\frac 1 n)$ to see that the limit is $(0.1)^{2}=0.01$.
set $1/n=h$

$$\sqrt{n^4-2n^3}=\sqrt{\dfrac{1-2h}{h^4}}=\dfrac{\sqrt{1-2h}}{h^2}$$

Rationalize the numerator

$$\lim\_{h\to0^+}\dfrac{\sqrt{1-2h}-1}{h(10+2h)}=\lim\_{h\to0^+}\dfrac{{1-2h}-1}{h(10+2h)(\sqrt{1-2h}+1)}=-\dfrac2{10(1+1)}$$