Question about linear machine learning models, positive coefficients, correlated features, and overfitting I am experimenting with a few different linear models from SKLearn. I am using a dataset with about 600 features and 350,000 samples. I have noticed that I get extreme overfitting unless I force the coefficients to be positive. Once I force coefficients to be positive I get a good match between in-sample and out of sample performance.
There is high correlation between the features, with many pairs above 0.7 measured by spearman correlation.
So my questions is whether the overfitting is due to the correlated features? Somehow when the algorithm is able to subtract two highly correlated features its able to overfit, but when it can only add them together it can't?

I think this may be possible.  Consider an extreme case where $x_1$ and $x_2$ come from the same data generating process but with independent noise terms.  And suppose $y=x_1+\epsilon$.  Then any model of the form $\hat{y}=\alpha x_1 + (1-\alpha)x_2$ is "correct", but by having the flexibility to choose $\alpha$, your model will select the one that best fits the additional noise.  By constraining to nonnegative coefficients, you constrain $\alpha\in[0,1]$, limiting the flexibility.
In this simple example that shouldn't lead to worse performance on unseen data (the model is "correct" after all), but I suspect a cleverer example could do it; maybe just with many features of a similar flavor, you give your model enough room to fit to some randomly present trend in the noise of the training set that isn't present in the test set?  On the other hand, your dataset size is large enough that I'm surprised you'd see such extreme overfitting just from this sort of effect, if the test set is randomly drawn.
