How many independet test-train splits (with independent training) should I perform? I recently read some literature by Bagnall et al https://arxiv.org/pdf/1602.01711.pdf "The Great Time Series Classification Bake Off". If I understand them correctly, they advise to perform multiple (independent) test-train splits to compare different models and their performance:

*

*Split the data into test train data

*Train a model on the training data from scratch

*Evaluate the model on the test data and obtain a performance measure

*Repeat, starting from (including) step 1 while

This procedure (equivalent to repeaded crossvalidation) then gives a distribution of performance measures.
While it is clear that this loop 1-4 is computationally demanding, we have the advantage of not relying on a single test-train split.
I wonder if there is any further literature on this or if there are known drawbacks with this method. Could we somehow run into overfitting with this procedure (even if the models in step 2 are guaranteed to be erased after evaluation of the performance) ?

This method of generating train/test splits is very similar to cross-validation, so you shouldn't have problems with overfitting - in fact one of the reasons it was used was to help prevent overfitting. One benefit over cross-validation is that the size of the test set is not dictated by the number of test folds. Which is useful in this case as it meant the authors could generate the same number of train/test splits for all the datasets used in the evaluation, even when there are fewer cases in the dataset than the required number of test folds.
The main draw-back is the one you've already mentioned - it requires more computing resources than (for example) a single train/test split.
As for the number of splits to use, "The Great Time Series Classification Bake Off" you reference used 100 splits. Other more recent papers proposing time series classifiers, such as Tan et al. "MultiRocket: multiple pooling operators and transformations for fast and effective time series classification", have used 30 splits.
