Best hyperparameter is not consistent among different seeds I do hyper-parameter tuning on my network and it outperforms the simple classifier. The difference in classification is considerable after hyper-parameter tuning. But, the problem is that an optimal hyper-parameter performs poorly in a different seed. In other words, I can not find an optimal hyper-parameter for all seeds. Is there any way I can do hyper-parameter tuning in a better way to find good hyper-parameters for all seeds?  Currently, I am using scikit-learn optimization package (skopt).


I can not find an optimal hyper-parameter for all seeds

That's the stochastic nature of machine learning. We don't have all the data, so any inference/training is vulnerable to noise. The model complexity, dataset size, your search, properties of the data all can affect this. Each seed is a separate snapshot that the real world represents. It's not expected to come up with the same set of optimal hyper-parameters for each seed. The most sensible way is the do multiple random experiments and use commonly selected hyper-parameters as your best and report accordingly; though not every research paper does that.
