@Oli Hey, thanks for your advice, I will try that first to see what happend. I check a lot of examples which call the validation function inside of the training process, none of them put the model back to train() again after validation. Thats why I am getting so confused.
Few people use cross-validation in deep learning, but it was very popular in other machine learning agorithums. I am not very familiar with that, it seems the only difference between k-fold and normal training, is the training set was divided into K subsets.