For positive real numbers $a, b, c$, maximise $(a-3)(b-2)(c-1)$ given $a+b+c=18$. For positive real numbers $a, b, c$, if $a + b + c = 18$, then max value of $(a-3)(b-2)(c-1) =$?
My approach:-
$a-3=x$
$b-2=y$
$c-1=z$
I get $x+y+z=12$ , and I need to find $\max(xyz)$.
So I applied AM-GM inequality here,
$\dfrac{x+y+z}{3} \geq \sqrt[3]{xyz}$,
which gives me the answer as $64$.
But the answer given in the book is $102$, and also it states that AM-GM cannot be applied here.  I don't understand why can't we apply it here, the only condition AM-GM requires is that we have positive real numbers, right, then what is the issue ?

The maximum value of $102$ can be proven if we relax the constraints slightly, such that $a,b,c$ are all non-negative reals, instead of positive reals.
Case $1$: All $3$ terms are greater than or equal to zero.
By AM-GM,
\begin{align}
\sqrt[3]{(a-3)(b-2)(c-1)} & \leq \dfrac{a-3+b-2+c-1}{3} \\
& =4.
\end{align}
This gives us $(a-3)(b-2)(c-1) \leq 64.$
Case $2$:
Exactly $1$ term is less than or equal to zero. Clearly, $(a-3)(b-2)(c-1) \leq 0.$
Case $3$:
All $3$ terms are less than or equal to zero, which is impossible since this would imply $a+b+c \leq 6$.
Case $4$:
Exactly $2$ terms are less than or equal to zero. There are $3$ further sub-cases to consider:
Sub-case $1$:
$a-3 \leq 0, \ c-1 \leq 0.$
Then,
\begin{align}
(a-3)(b-2)(c-1) & =(3-a)(b-2)(1-c) \\
\end{align}
It is obvious that $3-a \leq 3, \ b-2 \leq 16, \ 1-c \leq 1 \Rightarrow (3-a)(b-2)(1-c) \leq 48. $
Sub-case $2$:
$b-2 \leq 0, \ c-1 \leq 0.$
Then,
\begin{align}
(a-3)(b-2)(c-1) & =(a-3)(2-b)(1-c) \\
\end{align}
We have: $a-3 \leq 15, \ 2-b \leq 2, \ 1-c \leq 1 \Rightarrow (a-3)(2-b)(1-c) \leq 30. $
Sub-case $3$:
$a-3 \leq 0, \ b-2 \leq 0.$
Then,
\begin{align}
(a-3)(b-2)(c-1) & =(3-a)(2-b)(c-1) \\
\end{align}
We have: $3-a \leq 3, \ 2-b \leq 2, \ c-1 \leq 17 \Rightarrow (3-a)(2-b)(c-1) \leq 102. $ In particular, this maximum value is achieved when $a=0,\ b=0,\ c=18.$
