Is the magnitude coefficient vector in Ridge regression monotonic in lambda? recently an interesting question came up and while I would have intuitively said it is not, other students have now made a compelling case (while not being sure themselves).
For ridge (or l2 regularized) regression ( $RSS + \frac{\lambda}{2} ||w||^2 $)
Is the function of the magnitude of the coefficients $||w||^2 $ a function that is monotonic in lambda?
Thank you very much in advance for insights you can offer, this has been bugging me quite a bit this morning.

Another way of looking at it is that ridge regression is the Lagrangian for a related constrained optimisation problem, see my answer to a related question

Say we are optimizing a model with parameters $\vec{\theta}$, by
minimizing some criterion $f(\vec{\theta})$ subject to a constraint on
the magnitude of the parameter vector (for instance to implement a
structural risk minimization approach by constructing a nested
set of models of increasing complexity), we would need to solve:
$\mathrm{min}_\vec{\theta} f(\vec{\theta}) \quad \mathrm{s.t.} \quad \|\vec{\theta}\|^2 < C$
The Lagrangian for this problem is (caveat: I think, its been a long
day... ;-)
$\Lambda(\vec{\theta},\lambda) = f(\vec{\theta}) + \lambda\|\vec{\theta}\|^2 - \lambda C.$
So it can easily be seen that a regularized cost function is closely
related to a constrained optimization problem with the regularization
parameter $\lambda$ being related to the constant governing the
constraint ($C$), and is essentially the Lagrange multiplier.  The
$-\lambda C$ term is just an additive constant, so it doesn't change
the solution of the optimisation problem if it is omitted, just the
value of the objective function.
This illustrates why e.g. ridge regression implements structural risk
minimization: Regularization is equivalent to putting a constraint on
the magnitude of the weight vector and if $C_1 > C_2$ then every model
that can be made while obeying the constraint that
$\|\vec{\theta}\|^2 < C_2$
will also be available under the constraint
$\|\vec{\theta}\|^2 < C_1$.
Hence reducing $\lambda$ generates a sequence of hypothesis spaces of
increasing complexity.

So decreasing the ridge parameter corresponds to a loosening of the constraint on the squared norm of the weights, so if that was limiting the optimisation of the objective function, then the norm of the weights will increase.
