How many data points for test set in a time series I have a monthly sales data set from 2018 January onwards. I would like to know from expert what is the optimum train test split and minimum train test split. Also to mention that my data includes 2020 year data where the sales have been influenced due to pandemic, and 2021 have been recovering year.

*

*You want to have as many data points in the test set as possible.


*You also want to have as many data points in the training set as possible.
Given a constrain on the total amount of data points, increasing the data points in the test set will reduce the number of data points in the training set.
So you will want to look for an optimum.

Example/Illustration
Below is an example where we change training set size and test set size. We do this in order to see if there is some optimum for the sizes that is balancing the pros and cons.

*

*In the training we fit different models.

*In the testing we select the model which is best.

*We validate by computing the difference with the known theoretic function. (the theoretic function that we created ourselves, but we do the training and testing while pretending that we do not know the real theoretic function)

Let's use a model where we generate randomly 20 independent variables $X_i; 1\leq i \leq 20$ that are potentially in a linear model, which is in reality only made up from only the first five parameters $Y = X_1 + X_2 + X_3 + X_4 + X_5 + \text{noise}$. We do the fitting by regularization. Let's use 400 measurements for an experiment (which need to be divided into training and testing).
We repeat this experiment 100 times for each test set size.
Let's see how the regularization parameter varies based on the size of the test set.

Let's see how the performance parameter varies based on the size of the test set. (we plot the average of the 100 experiments per test size in red)

Interpretation of the graphs:

*

*In the first graph we see: when we increase the test size then this increases the lambda parameter. We can see this as an increase in the control of 'overfitting' (which is the reason that we use a test data set).


*In the second graph we see that there is some optimum for the test size. Increasing the test size initially improves the result, but at some point it does come with some costs. The reason is that increasing the test data size, decreases the training data size. When the training data size becomes smaller, then the models will perform less well.
In this particular example we see that this balance between large training data set (which improves the quality of the models) and the large test data set (which improves the control on overfitting) is optimal around a test data set of 33% of the entire data.

Note: The example indicates how the test data set size is influencing the results. But the result of 33% is not a general result. For each problem this size of the test data size differs. The optimum will depend on the amount of risks of overfitting versus risks of misfitting (noise).

R-code to produce graphs
library(glmnet)

set.seed(1)

n = 400
n_train = 20
vars = 20
lambda = 10^-4 * 1.5^c(1:60)

cvglmnet <- function(n_train) {
  
  ### create model matrix
  X <- matrix(rnorm(vars*n), ncol = vars)
  ### compute dependent variable 
  Y = rowSums(X[,1:5])+rnorm(n,0,10)
  
  # split data
  X_train = X[1:n_train,]
  Y_train = Y[1:n_train]
  X_test = X[(n_train+1):n,]
  Y_test = Y[(n_train+1):n]
  
  # perform model
  mod <- glmnet(X_train, Y_train, family = "gaussian", alpha = 0, lambda = lambda)
  
  ### compute mean train RSS
  Y_pred <- X_train %*% mod$beta
  errors <- Y_pred - Y_train
  RSS_train <- colSums(errors^2)/(n_train)
  
  ### compute mean test RSS
  Y_pred <- X_test %*% mod$beta
  errors <- Y_pred - Y_test
  RSS_test <- colSums(errors^2)/(n-n_train)
  
  ### the optimal model in the test 
  nm <- which.min(RSS_test)
  
  ### verify the model performance
  ### as distance in coefficients
  perf <- sum((mod$beta[,nm]-c(rep(1,5),rep(0,15)))^2)
  
  #plot(mod$beta[,nm])
  
  ### plotting
  #Ytot <- cbind(RSS_train,RSS_test)
  #plot(mod$lambda,RSS_train, log = "x", ylim = c(min(Ytot),max(Ytot)),
  #     ylab = "RSS", xlab = "lambda")
  #lines(mod$lambda,RSS_test)
  #points(mod$lambda[nm], RSS_test[nm], pch = 21, col = 1, bg = 2)
  
  return(list(lambda = mod$lambda[nm], performance = perf))
}

set.seed(1)
n_rep = 100
n_train <- seq(10,400,10)


### repeated computation of fitting procedure with different training/test data size
x <- replicate(n_rep,cvglmnet(n_train = n_train[1]))
Mlambda <- as.numeric(x[1,])
Mperformance <- as.numeric(x[2,])

for (i in 2:length(n_train)) {
  x <- replicate(n_rep,cvglmnet(n_train = n_train[i]))
  Mlambda <- cbind(Mlambda,as.numeric(x[1,])) 
  Mperformance <- cbind(Mperformance,as.numeric(x[2,])) 
}

plot(-1,-1, xlim = c(0,1), ylim = 10^c(-2,5), log = "y",
     xlab = "percent test size", ylab = "fitted lambda")

#plot(-1,-1, xlim = c(0,1), ylim = range(lambda), log = "y",
#     xlab = "percent test size", ylab = "fitted lambda")


for (i in 1:length(n_train)) {
  h <- Mlambda[,i]
  points(rep((n-n_train[i])/n,length(h)) + rnorm(length(h),0,0.01),h, pch = 21, 
         col = rgb(0,0,0,0.25), 
         bg = rgb(0,0,0,0.25), 
         cex = 0.7)
}



plot(-1,-1, xlim = c(0,1), ylim = c(0,15),
     xlab = "percent test size", ylab = "performance")

##plot(-1,-1, xlim = c(0,1), ylim = range(lambda), log = "y",
##     xlab = "percent test size", ylab = "fitted lambda")


for (i in 1:length(n_train)) {
  h2 <- Mperformance[,i]
  points(rep((n-n_train[i])/n,length(h2)) + rnorm(length(h2),0,0.01),h2, pch = 21, 
         col = rgb(0,0,0,0.25), 
         bg = rgb(0,0,0,0.25), 
         cex = 0.7)
  points((n-n_train[i])/n,mean(h2), pch = 21, col = 2, bg = 2)
}

 

