Why do you overfit if you train a linear regression model on a dataset that doesn't have enough datapoints? First of all, definitionally speaking, linear regressions tend to underfit (have high bias, low variance).
Additionally, just intuitively speaking, it seems like a linear regression would underfit in this situation. But the answer seems to be overfit.
Eg. If i have a dataset of only 3 points (clearly not enough), and i fit a linear regression on it, wouldn't you get a line that underfits? (see poorly drawn picture below that represents the linear "model" I'd imagine getting).
The only case where I could see a linear regression overfitting on these 3 points is if they happened to land perfectly in a linear line?


I find it totally plausible that your three points were generated by a downward-sloping trend and happened to have an especially high error term on the right-most point.
Maybe that’s not the most likely scenario, but that seems plausible, and if that is the case, your proposed regression line has overfit to an apparent upward trend.
If, instead of three points, you have ten or a thousand points, you get a better picture of the trend and are less likely to identify an upward trend when the true trend is downward.
SIMULATION
Let’s see how often an increasing relationship results in a negative regression slope for sample sizes of $3$, $10$, and $1000$.
set.seed(2023)
N1 <- 3
N2 <- 10
N3 <- 1000
R <- 1000
slopes1 <- slopes2 <- slopes3 <- rep(NA, R)
for (i in 1:R){
    
    x1 <- rnorm(N1)
    y1 <- x1 + rnorm(N1)
    L1 <- lm(y1 ~ x1)
    slopes1[i] <- summary(L1)$coef[2, 1]

    x2 <- rnorm(N2)
    y2 <- x2 + rnorm(N2)
    L2 <- lm(y2 ~ x2)
    slopes2[i] <- summary(L2)$coef[2, 1]

    x3 <- rnorm(N3)
    y3 <- x3 + rnorm(N3)
    L3 <- lm(y3 ~ x3)
    slopes3[i] <- summary(L3)$coef[2, 1]

}

ecdf(slopes1)(0) # 16% of slopes are negative for N = 3
ecdf(slopes2)(0) # 0.7% of slopes are negative for N = 10
ecdf(slopes3)(0) # No negative slopes for N = 1000

Analytically, it is the case that the standard errors of the regression coefficients shrink as the sample size increases, so the coefficient estimate variance decreases.
