Why are sklearn's cross_val_score values not increasing with the size of the training set? I am working on a lithology identification project similar to the one described here.
The idea is to train a model using well log data collected at a handful of drillholes, in order to predict the "rock type" in other drillholes.
I am comparing the performance of 3 different algorithms:

*

*AdaBoost

*SVM

*RandomForest

I am calling sklearn's cross_val_score to assess the performance of each model.
I did plot the mean score (cross_val_score(clf, X, y, cv=5).mean(), where X is the training data) for each method as a function of the number of holes used in my training set, i.e. the size of my training set.
Interestingly, the score does not increase as the size of the training set increases.

According to this stats stack exchange post:

one situation where more data does not help---and may even hurt---is
if your additional training data is noisy or doesn't match whatever
you are trying to predict.

Is this what is happening?
--- EDIT: answering the questions raised by @cbeleites unhappy with SX ---
My total dataset contains ~1,500 drill holes. For each hole, I have well log data from 4 different logs (let's call them measurement_A to measurement_D). Based on my knowledge of these well log data, I would say that they are not repeated measurements of the same feature, thus corresponding to what @cbeleites unhappy with SX calls situation (b) . These data are acquired using different sensors, have different units, etc... They essentially measure different mechanical properties, although in practice they often end up being correlated.
Out of these ~1,500 holes, I select 35 holes as my training set. The remainder of the data, I use as my test set.
In practice, my training_data consists of a Pandas dataframe containing the features on which to train the model, and the labels to predict.
Here is a simplified version of my train_model function
def train_model(training_data, features, labels):
    X=training_data[features]  # Features
    y=training_data[labels]  # Labels
    clf = make_pipeline(preprocessing.StandardScaler(), RandomForestClassifier(n_estimators=100, min_samples_split = 5, min_samples_leaf = 4, max_depth = 10, bootstrap = True))
    print(cross_val_score(clf, X, y, cv=5))
    classifier = clf.fit(X,y)
    return clf 

I call this function as follows:
clf = train_model(training_data, features_to_train_model, labels)

Where features_to_train_model = [measurement_A, measurement_B, measurement_C, measurement_D].
So in a nutshell, my X matrix contains 4 columns (one column per measurement) and 35,000 rows (one row per 1 cm depth increment, for all 35 drillholes), while my Y array contains 35,000 rows (again, one lithology label per depth).
I am aware that when I call cross_val_score(clf, X, y, cv=5), it does a test/train split on my training data only. I was naïvely expecting the scores to improve with an increasing dataset size, but that was neglecting the "error" referred to by @Sycorax.

I don't think this result is too surprising. Each of the points in your plot has an associated error measurement associated with it. The overall number of holes only varies in a small range, so the error will be similar at both extremes of number of holes. The largest number of holes you have is still quite small, so the results seem to be primarily driven by the scarcity of data. To apply a high-complexity model like boosting or random forest, you'll need much more data.
