How to deal with random parameters in MLOps I have a XGB model ready to go to production, in validation I discovered that the random seed makes reasonable difference in the performance of the model, which is pretty good, but for some seeds it's just good, and for others it's very good.
Now my intuition is that the random seed shouldn't make too much difference in a robust model, which probably means my model is overfitting, but if it is, it hasn't shown up in the other validation tests I've done.
Edit: extra info: this is a regression problem. I have used K-Fold CV to optimize hyperparameters, including alpha and gamma. The only part susceptible to randomness is the train/test split, so this tells me there's some kind of probability distribution in my data that may get better represented with some splits than with others. If this was a classification task, I could use stratified split to deal with this, but in this case, what's the correct approach? Finally, I haven't really set the seed anywhere, just ran the experiment 30 times and compared the results.

*

*Does this mean my model is overfitting? Should I try to regularize it and see if the random effect diminishes or disappears?


*When actually training the model that will go into production and on re-training in the future, how should I deal with the random seed? Optimizing it like a hyperparameter feels wrong to me, so should I just leave it at random? What's the recommended approach?

Your first question:
Since you are using the default setting of XGB, you are not using any built-in features to fight overfitting, so your model is probably indeed overfitting.
XGBoost provides two randomization techniques to fight overfitting, see the section "Control Overfitting" here.
Your second question:
Your setting of the seed will currently only effect your cross-validation (CV) splitting. IIUC, your variations in model performance refer to the different folds in CV. Once you have turned on the XGB internal randomization features, the overfitting should disappear and the results in CV should become similar.
CV is used for two purposes: first, to predict the error on the test dataset, and second, to use this predicted error to tune hyperparameters. But since you don't have any hyperparameters to tune (do you?), the only reason left to use CV is the prediction of the generalization error. But since you have a large dataset (100k), I recommend using just one single 80/20 partition, with training on the larger and testing on the smaller part. This should be totally sufficient to estimate your generalization capabilities, no need for CV. But, of course, you must make sure that your 80/20 partition is really random.
And then you leave the seed alone. Don't try to tune the seed, this is kind of a rule.
