How to split data as train and test set in a fixed manner? I've been struggling to figure out the best technique to assess model accuracy in relation to the train/test split.
Leave-one-out cross-validation and KFold appear to be more appropriate to utilize; however, LOOCV does not count more than one row of data as a holdout set, and KFold shuffles the data, preventing it from being in order.
Is there a suitable Python function for this?
Another point is, what is the right method for proving model accuracy?
Note: The data matrix is 900x1015, with one half (450x1015) devoted to one class and the other (450x1015) to another class. I employ two classification models: PCA and PLS-DA. Every three rows in each class contain one sample information, and the samples are in order. Besides, every three rows for one sample are the repeated measurements in different days.
The main picture can be like this;


Try out GroupKFold. It looks like it'll support what you need. If you don't already have a column that groups what you want together, you can make an additional column that identifies what to hold out, e.g. append a column [0,0,0,1,1,...,1] and specify that as your grouping separator. That'll separate your three rows (and sequences of three rows) from the rest of the data.
Check it out here
