Is the magnitude coefficient vector in Ridge regression monotonic in lambda? recently an interesting question came up and while I would have intuitively said it is not, other students have now made a compelling case (while not being sure themselves).
For ridge (or l2 regularized) regression ( $RSS + \frac{\lambda}{2} ||w||^2 $)
Is the function of the magnitude of the coefficients $||w||^2 $ a function that is monotonic in lambda?
Thank you very much in advance for insights you can offer, this has been bugging me quite a bit this morning.

Intuitive viewpoint
The regression is a trade-off in the terms $RSS$ and $||w||^2$, which are being balanced. If for some change in lambda the regression solution changes, then the one term increases while the other term decreases. They can't increase both because that will never lead to an improvement (and they can't decrease both because in the other direction that would mean an increase of both).
Now if you consider the sum $RSS(w) + 0.5 \lambda ||w||^2$ then you can see this as a sum of a decreasing and increasing (convex!) functions. As we increase $\lambda$ the minimum of the sum will shift into one direction.
Example:

library(glmnet)
set.seed(1)

x = matrix(rnorm(100 * 20), 100, 20)
y = rnorm(100)
fit1 = glmnet(x, y, alpha = 0)

beta = colSums(fit1$beta^2)
rss = (1-fit1$dev.ratio)*fit1$nulldev

plot(beta,rss, type ="l", main = "lowest/optimal RSS when norm of coefficients below beta", main.cex = 1) 
lines(beta, rss + 50*beta, col =2, lty = 2)
lines(beta, rss + 100*beta, col =2 ,lty = 2)
lines(beta, rss + 150*beta, col =2, lty = 2)

text(0.15,95.5, "relation RSS and beta")
text(0.08,99, "rss + 50 * beta" ,col = 2)
text(0.055,102, "rss + 100 * beta" ,col = 2)
text(0.04,103.5, "rss + 150 * beta" ,col = 2)

