k-fold cross validation and parameter tuning Say you want to train the hyperparameters of an RBF kernel in a regression context but you have a limited number of data. You do 5-fold cross-validation. You split your data into 5 different sets and you use 4 of them as the training set and one of them as the validation set. You do this 5 times. You report several metrics on each validation set such that the RMSE. In this way, you obtain 5 different values of hyperparameters of your RBF kernel. Now, assume that you have some more data coming in and you want to test the different values of hyperparameters you have. Which one do you select? The set of hyperparameters that gave you the lowest RMSE?  Moreover, to test your model on the test set, you train your model on the entire data set excluding the test set, right?

If you do 5-fold-CV, you get 5 different estimates of performance (in terms of your selected metric) - one per fold - for the same hyperparameters. I.e. you train your model 5 times with the same hyperparameters on 4 of the 5 splits of your data, each time leaving out one split of the data as validation. Then you calculate the metric of interest on the bit of data you left out.
If you then do the same thing again for different hyperparameters, you get another set of performance metrics. It can be useful to look at the mean metric value, but also the standard deviation of metrics across splits. There's different approaches to then pick the hyperparameters to use later. The "greedy" approach of picking the hyperparameters with the best mean metric value (or best mean + SD in case you want to favor models with less variable outcomes) may overfit the validation a bit, so there's various other ideas like the "1-standard-error rule (see e.g. Chapter 7 of "The elements of Statistical Learning" by Hastie, Tibshirani and Friedman, 2009). The 1-SE-rule says to pick the hyperparameters that lead to the most penalization/regularization (with multiple regularization parameters such an ordering is not always clear), but which still lie within 1 SE of the best mean metric value.
Other ideas could include model averaging or other ensembling techniques like stacking, which try to reflect the uncertainty around model choice by making use of all models to the extent that it seems helpful.
Retraining on all non-test data is indeed a sensible option. However, it can occasionally be difficult, e.g. if you need a validation set to determine when to stop training and the training process is too random so that you cannot just use the training length that worked during cross-validation. Other options include averaging the predictions of the models trained on the 5 different training data sets.
