Question: This is the first time I have attempted such a problem, and have been tackling it all morning. I've been trying to use simultaneous equations to solve for the variables, but it doesn't seem to be working out, so perhaps that's not the way to go...

$$\hat{\bf a} = \dfrac{1}{\sqrt{14}}({\bf i}+2{\bf j}+3{\bf k})$$
$$\hat{\bf b} = \dfrac{1}{\sqrt{5}}(2{\bf i}-{\bf j})$$
$$\hat{\bf c} = \dfrac{1}{\sqrt{70}}(3{\bf i}+6{\bf j}-5{\bf k})$$

Express the vector ${\bf d} = 5{\bf i}+3{\bf j}+2{\bf k}$ in the new $(\hat{\bf a},\hat{\bf b},\hat{\bf c})$ basis.

My attempted solution can be seen here: <https://drive.google.com/file/d/0B8CnEriPXOd1T3Z1amU4UGlMVU0/view>

There is basic arithmetic mistake in my attempted answer, so I have answered my own question below. Sorry for wasting anybody's time; I'll try to be more careful.

Answer: 
we will first deal with expressing as a linear combination of $\sqrt{14}\hat a, \sqrt 5\hat b,$ and $\sqrt{70}\hat c.$  

we need number $x, y, z$ so that $$x(i+2j+3k)+y(2i-j)+z(3i+6j-5k)=5i+3j+2k \tag 1 $$ 
in matrix form, we have $$\pmatrix{1&2&3&|&5\\2&-1&6&|&3\\3&0&-5&|&2}\to 
\pmatrix{1&0&0&|&1.21428\\0&1&0&|&1.40000\\0&0&1&|&0.32857}$$
the solutions $\pmatrix{x\\y\\z}$ is the last column in the second matrix. now, multiplying $x,y,z$ by $\sqrt{14}, \sqrt{5}, \sqrt{70}$ gives you the answer.
so you are looking for [$\alpha, \beta, \gamma$] where $\vec b = \alpha \hat a + \beta \hat b + \gamma \hat c =5\hat i+3\hat j+2\hat k$.

By rules of vectors, if you project $\vec b$ onto $\hat i$, you get the value of that unit vector only, which would be 5 in this case.

mathmatically it looks like this

$\vec d ⋅\hat i = ( 5\hat i\hat i ) + ( 2\hat j\hat i) + ( 3\hat k\hat i )$

where $\hat i\hat i = 1$ and $ \hat j\hat i = \hat k\hat i = 0$

so,
$\vec d ⋅\hat i = 5 \times 1 = 5$

we can then do the same with each unit vector you have in your new basis

$\vec d ⋅\hat a = \frac{1}{\sqrt14}(5\times1 + 3\times2 + 2\times3) =\frac{17}{\sqrt14} = \alpha$

$\vec d ⋅\hat b = \frac{1}{\sqrt5}(5\times2 + 3\times-1 + 2\times0) =\frac{7}{\sqrt5} = \beta$

$\vec d ⋅\hat c = \frac{1}{\sqrt70}(5\times3 + 3\times6 + 2\times-5) = \frac{23}{\sqrt70} = \gamma$

Finally we can say that vector $\vec b$ with unit basis $(\hat a, \hat b, \hat c)$ is equal to

$\vec d = \frac{17}{\sqrt14} \hat a + \frac{7}{\sqrt5} \hat b + \frac{23}{\sqrt70} \hat c$