Reducing training non-determinism and doing n-fold cross-validation is pretty reliable. But all such stuff is obviously slow…



 AlperEkmekci:

Additionally when we use different criterion loss functions, how can we choose one and use one of them and be sure about which one is the better than the other ?


If you switch a loss function, you postulate a different probability distribution family for predicted values. If that’s your intent, it is possible to compare model log-likelihoods (for non-deep compact models that would be AIC). Simpler approach is just comparing some auxiliary metrics.