How to deal with random parameters in MLOps I have a XGB model ready to go to production, in validation I discovered that the random seed makes reasonable difference in the performance of the model, which is pretty good, but for some seeds it's just good, and for others it's very good.
Now my intuition is that the random seed shouldn't make too much difference in a robust model, which probably means my model is overfitting, but if it is, it hasn't shown up in the other validation tests I've done.
Edit: extra info: this is a regression problem. I have used K-Fold CV to optimize hyperparameters, including alpha and gamma. The only part susceptible to randomness is the train/test split, so this tells me there's some kind of probability distribution in my data that may get better represented with some splits than with others. If this was a classification task, I could use stratified split to deal with this, but in this case, what's the correct approach? Finally, I haven't really set the seed anywhere, just ran the experiment 30 times and compared the results.

*

*Does this mean my model is overfitting? Should I try to regularize it and see if the random effect diminishes or disappears?


*When actually training the model that will go into production and on re-training in the future, how should I deal with the random seed? Optimizing it like a hyperparameter feels wrong to me, so should I just leave it at random? What's the recommended approach?

This is very strange. I never had any experience on the random seed being important. Specially in XBG which has no random component (in the default configuration) as far as I can remember.
XGB can implement random forests, and they are probably more sensitive to random initialization, but even than it should not make a "reasonable difference".
Maybe you can post details of the parametrization of the XBG and also details of the dataset - specially proportion of the classes. The only source of problems I can think now is a dataset with very few examples of some classes, and in some internal sampling of the training set (with as far as I know XGB does not do by default - parameter sampling_method) some sets are left without examples of these minority classes.
@Lerner Zhang link for reproducibility is very interesting.
Finally, there is no "random seed search". there is no structure for the random seed search - if a seed of 42 yields a good result, a seed of 43 may yield a bad result, and 44 an even better result. There is no meaningful search. I think the usual practice is to fix the random seed always with a known seed - at least you can reproduce the results!
