What is the structure of a GAM fit with `mgcv::bam()` with the `rho` parameter set? If I fit the model mgcv::bam(y~s(x), rho=0.8), where y and x are ordered by time, my understanding is that the model can be described in math notation as:
$
y_t = \beta0+f(x_t)+\epsilon_t\\
\epsilon_t = rho * \epsilon_{t-1} + w_t\\
w_t \sim normal(0, \sigma)
$
Is this correct?
Also is the maximum likelihood estimate of the model equivalent to minimizing the square of $w$?

I tend to think of the model mgcv::bam(y ~ s(x), rho = 0.8) as being equivalent to
$$
y_t = \beta_0 + f(x_t) + \varepsilon_t
$$
where $\boldsymbol{\varepsilon} \sim \mathcal{N}(0, \hat{\sigma}^2\boldsymbol{\Lambda})$ and $\boldsymbol{\Lambda})$ is the first order correlation matrix
$$
\boldsymbol{\Lambda} = \begin{pmatrix}
1          & \rho       & \rho^2     & \cdots & \rho^{n-1} \\
\rho       & 1          & \rho       & \cdots & \rho^{n-2} \\
\rho^2     & \rho       & 1          & \cdots & \rho^{n-3} \\
\vdots     & \vdots     & \vdots     & \ddots & \vdots     \\
\rho^{n-1} & \rho^{n-2} & \rho^{n-3} & \cdots & 1          \\
\end{pmatrix}
$$
with $\rho = 0.8$.
But this arises from the assumed first order AR process
$$\varepsilon_t = \rho \varepsilon_{t-1} + \eta_t$$
with $\eta_t \sim \mathcal{N}(0, \hat{\sigma}^2)$. (Which is what you wrote but with the notation from my own lecture notes/slides so as to not confuse myself.)
This is essentially a generalised least squares problem, which I understand will seek to minimise the squared Mahalanobis length of the residual vector $\eta_t$ (your $w_t$).
