When I come across implementations of ImageNet I always see people splitting validation and training set into discrete folders before data is loaded. However this is problematic if one wants to apply cross validation to his model. If I have a small test set the estimate would not be trustworthy unless I apply a CV method. This would mean training the whole model again with different train and test set in each iteration. Is there a more flexible alternative to using folders for train / test split?