5. As shown in Figure 1, in a triangular prism, it is known that the plane determined by points $A$, the midpoint $M$ of $B B_{1}$, and the midpoint $N$ of $B_{1} C_{1}$ divides the triangular prism into two parts with different volumes. The ratio of the volume of the smaller part to the volume of the original triangular prism is ( ).
(A) $\frac{23}{36}$
(B) $\frac{13}{36}$
(C) $\frac{13}{23}$
(D) $\frac{12}{23}$
5. B.

As shown in Figure 4, connect $NM$, intersecting the extensions of $CB$ and $CC_{1}$ at points $D$ and $E$, respectively, and connect $AD$.

According to the problem, point $E$ is also in the plane determined by points $A$, $M$, $N$, and $D$.

Let the plane $AMN$ intersect $A_{1}C_{1}$ at point $F$. Let the base area of the original triangular prism be $S$, and the height be $h$. The plane divides the original triangular prism into two parts, with volumes denoted as $V_{1}$ and $V_{2}$, respectively.
Since $M$ is the midpoint of $BB_{1}$, we have
$\triangle B_{1}NM \cong \triangle BDM \Rightarrow \frac{DB}{BC}=\frac{1}{2}$.
Thus, $S_{\triangle ADC}=\frac{3}{2} S$.
Similarly, $\frac{EC_{1}}{C_{1}C}=\frac{1}{2}$.
$$
\begin{array}{l}
=\frac{1}{3} \times \frac{3}{2} S \times \frac{3}{2} h-\frac{1}{3}\left(\frac{3}{2} S \times \frac{1}{9}\right) \times \frac{1}{2} h- \\
\frac{1}{3} \times \frac{S}{2} \times \frac{1}{2} h \\
=\frac{23}{36} S h, \\
V_{1}=S h-\frac{23}{36} S h=\frac{13}{36} S h. \\
\end{array}
$$

Therefore, the ratio of the volume of the smaller part to the volume of the original triangular prism is
$$
\text { is } \frac{V_{1}}{V}=\frac{13}{36} \text {. }
$$