5. Given that the three vertices of isosceles right $\triangle P Q R$ lie on the three sides of isosceles right $\triangle A B C$. Then the minimum value of $\frac{S_{\triangle P Q R}}{S_{\triangle A B C}}$ is ( ).
(A) $\frac{1}{2}$
(B) $\frac{1}{3}$
(C) $\frac{1}{4}$
(D) $\frac{1}{5}$
5. D.

Consider two cases.
(1) When the right-angle vertex of $\triangle P Q R$ is on the hypotenuse of $\triangle A B C$, as shown in Figure 2. Then
$$
\begin{array}{l}
P, C, Q, R \text { are concyclic } \\
\Rightarrow \angle A P R=\angle C Q R \\
=180^{\circ}-\angle B Q R \\
\Rightarrow \sin \angle A P R \\
=\sin \angle B Q R .
\end{array}
$$

Applying the Law of Sines in $\triangle A P R$ and $\triangle B Q R$ respectively, we get
$$
\begin{array}{l}
\frac{P R}{\sin A}=\frac{A R}{\sin \angle A P R}, \\
\frac{Q R}{\sin B}=\frac{B R}{\sin \angle B Q R} .
\end{array}
$$

Since $\angle A=\angle B=45^{\circ}$, it follows that $P R=Q R$.
Thus, $A R=B R$, meaning $R$ is the midpoint of side $A B$.
Draw $R H \perp A C$ at point $H$.
Then $P R \geqslant R H=\frac{1}{2} B C$
$$
\Rightarrow \frac{S_{\triangle P Q R}}{S_{\triangle A B C}}=\frac{P R^{2}}{B C^{2}} \geqslant \frac{\left(\frac{1}{2} B C\right)^{2}}{B C^{2}}=\frac{1}{4} \text {. }
$$

In this case, the minimum value of $\frac{S_{\triangle P Q R}}{S_{\triangle A B C}}$ is $\frac{1}{4}$.
(2) When the right-angle vertex of $\triangle P Q R$ is on one of the legs of $\triangle A B C$, as shown in Figure 3.
Let $B C=1$,
$$
\begin{array}{l}
C R=x(0 \leqslant x<1), \\
\angle B R Q \\
=\alpha\left(0<\alpha<\frac{\pi}{2}\right) .
\end{array}
$$

Then $\angle C P R$
$$
\begin{aligned}
& =90^{\circ}-\angle P R C \\
= & \angle B R Q=\alpha
\end{aligned}
$$

In the right triangle $\triangle C P R$,
$$
P R=\frac{C R}{\sin \alpha}=\frac{x}{\sin \alpha} .
$$

In $\triangle B R Q$, by the Law of Sines, we have
$$
\begin{array}{l}
\frac{R Q}{\sin B}=\frac{R B}{\sin \angle R Q B} \\
\Leftrightarrow \frac{\frac{x}{\sin \alpha}}{\sin \frac{\pi}{4}}=\frac{1-x}{\sin \left(\frac{3 \pi}{4}-\alpha\right)} \\
\Leftrightarrow \frac{x}{\sin \alpha}=\frac{1}{\cos \alpha+2 \sin \alpha} .
\end{array}
$$

Then $S_{\triangle P Q R}=\frac{1}{2} P R^{2}=\frac{1}{2}\left(\frac{x}{\sin \alpha}\right)^{2}$
$$
\begin{array}{l}
=\frac{1}{2}\left(\frac{1}{\cos \alpha+2 \sin \alpha}\right)^{2} . \\
\text { Hence } \frac{S_{\triangle P Q R}}{S_{\triangle A B C}}=\left(\frac{1}{\cos \alpha+2 \sin \alpha}\right)^{2} \\
\geqslant \frac{1}{\left(1+2^{2}\right)\left(\cos ^{2} \alpha+\sin ^{2} \alpha\right)}=\frac{1}{5},
\end{array}
$$

with equality holding if and only if $\alpha=\arctan 2$. In this case, the minimum value of $\frac{S_{\triangle P Q R}}{S_{\triangle A B C}}$ is $\frac{1}{5}$.