How to split data as train and test set in a fixed manner? I've been struggling to figure out the best technique to assess model accuracy in relation to the train/test split.
Leave-one-out cross-validation and KFold appear to be more appropriate to utilize; however, LOOCV does not count more than one row of data as a holdout set, and KFold shuffles the data, preventing it from being in order.
Is there a suitable Python function for this?
Another point is, what is the right method for proving model accuracy?
Note: The data matrix is 900x1015, with one half (450x1015) devoted to one class and the other (450x1015) to another class. I employ two classification models: PCA and PLS-DA. Every three rows in each class contain one sample information, and the samples are in order. Besides, every three rows for one sample are the repeated measurements in different days.
The main picture can be like this;



LOOCV does not count more than one row of data as a holdout set,

that is what most ready-to-use implementations do by default. However, you could do leave-one-sample-out-CV.

one half (450x1015) devoted to one class and the other (450x1015) to another class

if you want to preserve the 50:50 relative class frequencies, use stratification

every three rows for one sample are the repeated measurements in different days.

Here it is not yet clear whether random factor measurement (repetition) is nested within random factor sample (patient) or crossed, and whether measurement day is a fixed or random factor. This will depend on your precise application scenario, but correct splitting will depend on this.

*

*For random measurement days nested within sample, split at the uppermost
level of the nesting, i.e. sample (as in the question), GroupKFold() or StratifiedGroupKFold() provide this.


*For random measurement days crossed with sample, you need to produce test sets which are independent both in sample and in measurement day.
You'll likely set up the splitting yourself and then hand it over to PredefinedSplit()


*Other application scenarios (e.g. predict second/third (fixed) measurement from first one/two -> time series treatment) have yet different needs.
