11. In Rt $\triangle ABC$, $\angle ACB=90^{\circ}, BC=a, CA=b(a>0, b>0)$, point $P$ is on side $AB$. Fold Rt $\triangle ABC$ along line $PC$ to form a tetrahedron $PABC$. Try to find the maximum possible volume of the tetrahedron.
【Answer】 $\frac{1}{6} \cdot \frac{a^{2} b^{2}}{\left(a^{\frac{2}{3}}+b^{\frac{2}{3}}\right)^{\frac{3}{2}}}$

【Detailed Solution】

Let the volume of tetrahedron $P A B C$ be $V$.
As shown in the figure, draw $P D \perp A C$ at point $D$, and $B E \perp P C$ at point $E$. Then $V \leq \frac{1}{3}\left(\frac{1}{2} P D \cdot A C\right) \cdot B E$ $=\frac{b}{6} P D \cdot a \sin \theta=\frac{a b}{6} P D \sin \theta=\frac{a b}{6} P D \sin \left(90^{\circ}-\varphi\right)=\frac{a b}{6} P D \cos \varphi$.
The equality holds if and only if the plane $P C B \perp$ plane $P C A$.
In $\triangle P A C$, by $A D+D C=b$, we get $P D \cot \alpha+P D \cot \varphi=b$, where $\cot \alpha=\frac{b}{a}$, then $P D=\frac{b}{\frac{b}{a}+\cot \varphi}$,
$$
V \leq \frac{a b}{6} \cdot \frac{b \cos \varphi}{\frac{b}{a}+\cot \varphi}=\frac{a^{2} b^{2}}{6} \cdot \frac{\cos \varphi}{b+a \cot \varphi}=\frac{a^{2} b^{2}}{6} \sqrt{\frac{\cos ^{2} \varphi}{(b+a \cot \varphi)^{2}}}=\frac{a^{2} b^{2}}{6} \cdot \frac{1}{\sqrt{\left(1+\tan ^{2} \varphi\right)(b+a \cot \varphi)^{2}}} .
$$

Let $x=\tan \varphi(x>0)$,
$$
f(x)=\left(1+x^{2}\right)\left(b+\frac{a}{x}\right)^{2}=b^{2} x^{2}+2 a b x+a^{2}+b^{2}+\frac{2 a b}{x}+\frac{a^{2}}{x^{2}}=\left(b^{2} x^{2}+\frac{a b}{x}+\frac{a b}{x}\right)+\left(a b x+a b x+\frac{a^{2}}{x^{2}}\right)+a^{2}+b^{2}
$$
$\geq 3 \sqrt[3]{a^{2} b^{4}}+3 \sqrt[3]{a^{4} b^{2}}+a^{2}+b^{2}=\left(a^{\frac{2}{3}}+b^{\frac{2}{5}}\right)^{3}$. The equality holds if and only if $x=\sqrt[3]{\frac{a}{b}}$.
$$
V \leq \frac{a^{2} b^{2}}{6} \cdot \frac{1}{\sqrt{f(x)}} \leq \frac{a^{2} b^{2}}{6} \cdot \frac{1}{\left(a^{\frac{2}{3}}+b^{\frac{2}{3}}\right)^{\frac{3}{2}}}=\frac{1}{6} \cdot \frac{a^{2} b^{2}}{\left(a^{\frac{2}{3}}+b^{\frac{2}{3}}\right)^{\frac{3}{2}}} \text {. }
$$

The equality holds if and only if the plane $P C B \perp$ plane $P C A$, and $\tan \varphi=\left(\frac{a}{b}\right)^{\frac{1}{3}}$. Therefore, the maximum volume of the tetrahedron is $\frac{1}{6} \cdot \frac{a^{2} b^{2}}{\left(a^{\frac{2}{3}}+b^{\frac{2}{3}}\right)^{\frac{3}{2}}}$.