The idea of k-fold is to show, that your model generalizes well on different subsets of your data. You can use it to have a relatively large trainsets. After you ensured that your model generalizes well, you might want to retrain the network on all your data (without k-fold splits) to get the best network possible by using all available training data (usually resulting in a more robust model).