5. In $\triangle A B C$, $A B=B C>A C, A H$ and $A M$ are the altitude and median from vertex $A$ to side $B C$, respectively, and $\frac{S_{\triangle A M H}}{S_{\triangle A B C}}=\frac{3}{8}$. Determine the value of $\cos \angle B A C$.
5. In $\triangle A B C$, draw $B D \perp A C$ at point $D$. Let $A B=A C=a$.
Since $\frac{S_{\triangle M M H}}{S_{\triangle A B C}}=\frac{M H}{B C}=\frac{3}{8}$, therefore,
$M H=\frac{3}{8} a$,
$H C=M C-M H=\frac{1}{2} a-\frac{3}{8} a=\frac{1}{8} a$.
Let $\angle B A C=\beta$. Then $\angle B C A=\beta$.
In Rt $\triangle B C D$, $D C=a \cos \beta$.
In $\triangle A B C$, $A C=2 D C=2 a \cos \beta$.
In Rt $\triangle A C H$,
$\cos \beta=\frac{H C}{A C}=\frac{\frac{1}{8} a}{2 a \cos \beta} \Rightarrow \cos ^{2} \beta=\frac{1}{16}$.
Since $\beta$ is an acute angle, thus, $\cos \beta=\frac{1}{4}$.