12. Given $f(x)=\ln (a x+b)+x^{2}(a \neq 0)$.
(1) If the tangent line to the curve $y=f(x)$ at the point $(1, f(1))$ is $y=x$, find the values of $a$ and $b$;
(2) If $f(x) \leqslant x^{2}+x$ always holds, find the maximum value of $a b$.
(1) $\left\{\begin{array}{l}f^{\prime}(1)=\frac{a}{a+b}+2=1, \\ f(1)=\ln (a+b)+1=1\end{array} \Rightarrow\left\{\begin{array}{l}\frac{a}{a+b}=-1, \\ a+b=1\end{array} \Rightarrow\left\{\begin{array}{l}a=-1, \\ b=2 .\end{array}\right.\right.\right.$
(2) $f(x) \leqslant x^{2}+x \Rightarrow \ln (a x+b)-x \leqslant 0$,

Let $g(x)=\ln (a x+b)-x, g^{\prime}(x)=\frac{a}{a x+b}-1=\frac{-a x+a-b}{a x+b}$, according to the problem, $a>0, a-b>0$, thus $g(x)$ is monotonically increasing on $\left(-\frac{b}{a}, 1-\frac{b}{a}\right)$ and monotonically decreasing on $\left(1-\frac{b}{a},+\infty\right)$,
Therefore, $g\left(1-\frac{b}{a}\right)=\ln a+\frac{b}{a}-1 \leqslant 0 \Rightarrow b \leqslant a-a \ln a \Rightarrow a b \leqslant a^{2}-a^{2} \ln a$.
Let $h(x)=x^{2}-x^{2} \ln x, h^{\prime}(x)=2 x-2 x \ln x-x=x(1-2 \ln x)$, then $h(x)$ is monotonically increasing on $(0, \sqrt{\mathrm{e}})$ and monotonically decreasing on $(\sqrt{\mathrm{e}},+\infty)$, so $h(x)_{\max }=h(\sqrt{\mathrm{e}})=\frac{\mathrm{e}}{2}$.
In summary, when $a=\sqrt{\mathrm{e}}, b=\frac{\sqrt{\mathrm{e}}}{2}$, the maximum value of $a b$ is $\frac{\mathrm{e}}{2}$.