Find a vector that bisects the smaller of the two angles formed by vectors <3,4> and <5,-12>. The solution is <8,-1>. I tried finding the angle between the two vectors, but wasn't sure what to do next. 

The cosines of the angles have to be the same and positive (for the smaller angle). Therefore 
$$\frac{3 a + 4 b}{\sqrt{3^2 + 4^2} \cdot \sqrt {a^2 + b^2} } = \frac{5 a + (-12) b}{\sqrt{5^2 + (-12)^2} \cdot \sqrt {a^2 + b^2} }>0$$ or
$$\frac{3 a + 4 b}{5 } = \frac{5 a + (-12) b}{13 }>0$$
Check that $(a,b) = (8,-1)$ is a solution. Indeed:
$$\frac{3 \cdot 8 + 4 \cdot (-1)}{5}= 4$$ and
$$\frac{5 \cdot 8 + (-12) \cdot (-1)}{13}=4$$
