Cross-validation by hand in R Trying to teach myself cross-validation on a super simple example, linear regression. My understanding is that when I build a model via CV, it should have a lower RMSE. But I find almost identical RMSEs with and without CV. Is this because I've coded too simple a problem? Or because I've implemented CV incorrectly?
Here is the code:
library(tidyverse)

set.seed(1234)
n = 1e4
x = rnorm(n) 
y = 1 + 3*x + rnorm(n)
df = data.frame(y,x)
df$id = 1:nrow(df)
df$train = sample(c(TRUE, FALSE), nrow(df), replace=TRUE, prob=c(0.7,0.3))
df_train = subset(df, train)
df_test = subset(df, !train)
k = 10 # 10 folds
df_train$fold = sample(rep(1:k, each = nrow(df_train)/k))

results = vector(mode = 'list', length = k)
for(i in 1:k){
  cv_train = df_train[df_train$fold != i,]
  cv_test = df_train[df_train$fold == i,]
  m = lm(y ~ x, data = cv_train)
  cv_test$y_resid = cv_test$y - predict(m, newdata = cv_test)
  results[[i]] = cv_test
  rm(cv_train, cv_test, m)
}
df2 = bind_rows(results)
rmse_by_fold = df2 %>% group_by(fold) %>% summarise(rmse = sqrt(mean(y_resid**2)))
# rmse
mean(rmse_by_fold$rmse)


# no cross validation
m = lm(y~x, data = df_train)
y_resid = df_train$y - predict(m, newdata = df_train)
# rmse
sqrt(mean(y_resid**2))


There is a design flaw here: you are fitting the true model. To be precise, you simulated (x, y) from a straight line and fit a straight line. You will "perfectly" recover the truth, so you will observe that test error is about as same as training error.
Cross-Validation is used to combat overfitting. That is, you specify a model that is more complicated than the ground truth. For example, if the ground truth is a straight line but you train a quintic polynomial, then you will observe a small training error but a high test error.
When you have a sequence of candidate models, Cross-Validation can be used for model selection. Suppose the ground truth is a cubic polynomial, and you have 6 candidate models, namely polynomials of degree 1 to 6. Then you can compute a CV score for each choice of degree, and plot CV score against degree. You shall see that degree = 3 gives the minimal CV error.
