@mMagmer thanks it works, though I am getting a bit of higher loss (than without the k-fold validation model)
Epoch:45/50 AVG Training Loss:1.538 AVG valid Loss:1.521 %
Epoch:46/50 AVG Training Loss:1.493 AVG valid Loss:1.605 %
Epoch:47/50 AVG Training Loss:1.473 AVG valid Loss:1.450 %
Epoch:48/50 AVG Training Loss:1.512 AVG valid Loss:1.462 %
Epoch:49/50 AVG Training Loss:1.429 AVG valid Loss:1.656 %
Epoch:50/50 AVG Training Loss:1.421 AVG valid Loss:1.400 %

Out of curiosity, I would like to know, say I saved a model, and then if I test the model with the same test dataset multiple times, then the prediction (prediction/model output) will be the same for the same test dataset?
For example, the test dataset image and the target is 7, and the prediction from the model is 6 (first time). At the second time, if I give the same image and target (the previous one) then the output will be 6 (as I am using the same model)?