<p dir="auto">ROC-AUC SVM classification results for random, normally distributed data does not produce a <em>symmetric</em> distribution around 0.5. A tail of high ROC-AUC scores can be observed:<br>
<a target="_blank" rel="noopener noreferrer nofollow" href="https://user-images.githubusercontent.com/80628104/112497896-510ef180-8d86-11eb-9bf5-2b0f3b65dd9d.png"><img src="https://user-images.githubusercontent.com/80628104/112497896-510ef180-8d86-11eb-9bf5-2b0f3b65dd9d.png" alt="rng" style="max-width: 100%;"></a></p>
<p dir="auto">You can find more information in the Discussion thread (<a href="https://github.com/scikit-learn/scikit-learn/discussions/19712" data-hovercard-type="discussion" data-hovercard-url="/scikit-learn/scikit-learn/discussions/19712/hovercard">link</a>). Here is the code to reproduce the result:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import numpy as np
from sklearn.datasets import make_classification
from sklearn.feature_selection import SelectKBest
from sklearn.model_selection import GridSearchCV, RepeatedStratifiedKFold
from sklearn.metrics import roc_auc_score
from sklearn.pipeline import Pipeline
from sklearn.svm import SVC
import matplotlib.pyplot as plt
import itertools

r = 1
# Data-set dimensions and imbalance to match original data.
X, y = make_classification(n_samples = 105, n_features = 24, weights = [27/105], random_state = r)
scores = list()
for rng in range(100):
    X = np.random.normal(size = X.shape)
    clf = Pipeline([('anova', SelectKBest()), ('svc', SVC(kernel = 'linear'))])

    # Reduced grid-search for convenience
    K = [1, 2, 5]   
    C = [0.1, 1]
   
    space = dict()
    space['anova__k'] = K
    space['svc__C'] = C
    
    scores_ = list()
    cv_out = RepeatedStratifiedKFold(n_splits = 10, n_repeats = 10, random_state = r)
    for train_indx, test_indx in cv_out.split(X, y):
        X_train, y_train = X[train_indx, :], y[train_indx]
        X_test, y_test = X[test_indx, :], y[test_indx]        

        cv = RepeatedStratifiedKFold(n_splits = 5, n_repeats = 5, random_state = r)
        search = GridSearchCV(clf, space, scoring = 'roc_auc', cv = cv, refit = True, n_jobs = -1)
        result = search.fit(X_train, y_train)
        best_model = result.best_estimator_
        
        y_pred = best_model.decision_function(X_test)
    
        scores_.append(roc_auc_score(y_test, y_pred))
        
    scores.append(scores_)
    print('Rng iteration:', rng + 1, '/ 100')

plt.hist(np.mean(np.array(scores), axis = 1))
plt.xlabel('ROC-AUC')
plt.show()"><pre class="notranslate"><code class="notranslate">import numpy as np
from sklearn.datasets import make_classification
from sklearn.feature_selection import SelectKBest
from sklearn.model_selection import GridSearchCV, RepeatedStratifiedKFold
from sklearn.metrics import roc_auc_score
from sklearn.pipeline import Pipeline
from sklearn.svm import SVC
import matplotlib.pyplot as plt
import itertools

r = 1
# Data-set dimensions and imbalance to match original data.
X, y = make_classification(n_samples = 105, n_features = 24, weights = [27/105], random_state = r)
scores = list()
for rng in range(100):
    X = np.random.normal(size = X.shape)
    clf = Pipeline([('anova', SelectKBest()), ('svc', SVC(kernel = 'linear'))])

    # Reduced grid-search for convenience
    K = [1, 2, 5]   
    C = [0.1, 1]
   
    space = dict()
    space['anova__k'] = K
    space['svc__C'] = C
    
    scores_ = list()
    cv_out = RepeatedStratifiedKFold(n_splits = 10, n_repeats = 10, random_state = r)
    for train_indx, test_indx in cv_out.split(X, y):
        X_train, y_train = X[train_indx, :], y[train_indx]
        X_test, y_test = X[test_indx, :], y[test_indx]        

        cv = RepeatedStratifiedKFold(n_splits = 5, n_repeats = 5, random_state = r)
        search = GridSearchCV(clf, space, scoring = 'roc_auc', cv = cv, refit = True, n_jobs = -1)
        result = search.fit(X_train, y_train)
        best_model = result.best_estimator_
        
        y_pred = best_model.decision_function(X_test)
    
        scores_.append(roc_auc_score(y_test, y_pred))
        
    scores.append(scores_)
    print('Rng iteration:', rng + 1, '/ 100')

plt.hist(np.mean(np.array(scores), axis = 1))
plt.xlabel('ROC-AUC')
plt.show()
</code></pre></div>
<p dir="auto">I am not sure whether this result is to be expected, is due to a flaw in my code, or something else altogether. It might just as well be a trivial matter and I apologize if this is not the right place to ask: prior to here and in addition to the Discussion thread linked above, I have also asked this question in Stack Overflow (<a href="https://stackoverflow.com/questions/66734960/biased-svm-classification-results-for-random-data" rel="nofollow">link</a>), r/learnpython (<a href="https://www.reddit.com/r/learnpython/comments/mbc0it/biased_svm_classification_results_for_random_data/" rel="nofollow">link</a>) and r/scikit_learn (<a href="https://www.reddit.com/r/scikit_learn/comments/mczikb/biased_svm_classification_results_for_random_data/" rel="nofollow">link</a>) over the past week, receiving no answer.</p>
<p dir="auto">Please let me know if there is anything I can do to help receiving an answer.</p>
<p dir="auto">LSHForest should be deprecated and scheduled for removal in 0.21. It should also warn about having bad performance. cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ogrisel/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ogrisel">@ogrisel</a></p>