Using the mean of predict_proba outputs as an indicator of potential classifier accuracy for semi supervised learning Compare these code examples:
import lightgbm, numpy as np, pandas as pd
d = pd.DataFrame(np.random.randint(10, size=(10000,100)))
d['y'] = d.apply(lambda x: x[0] if np.random.random() < 0.55 else np.random.randint(10), axis = 1)
cl = lightgbm.LGBMClassifier().fit(d.iloc[:9000,0:10], d[:9000]['y'])
print(cl.score(d.iloc[9000:,0:10], d[9000:]['y']))
pr = cl.predict_proba(d.iloc[9000:,0:10])
print(np.mean([x.max() for x in pr]))



d = pd.DataFrame(np.random.randint(10, size=(10000,100)))
d['y'] = d.apply(lambda x: x[0] if np.random.random() < 0.95 else np.random.randint(10), axis = 1)
cl = lightgbm.LGBMClassifier().fit(d.iloc[:9000,0:10], d[:9000]['y'])
print(cl.score(d.iloc[9000:,0:10], d[9000:]['y']))
pr = cl.predict_proba(d.iloc[9000:,0:10])
print(np.mean([x.max() for x in pr]))

output
 0.59
 0.6095001350341738
 0.956
 0.9934019651423959

What I'm doing is training lgbm classifier with data that is very noisy, and taking the mean of the max predict_proba values.  I then do it again with data is that is only slightly noisy.
The mean of the predict proba max goes way up for the slightly noisy versus the very noisy data.
Is this a reasonable method of predicting the accuracy of a classifier without access to the labels?  I can't find any literature on this seemingly very important topic.  Papers, terminology, etc welcomed.
I understand this is a type of semi supervised learning, but can't find anything talking about this evaluation metric in particular.  I also understand that this might only roughly work under certain assumptions.  What would those assumptions be?

Looking at your code, you have:

*

*Created a dataset where the label d['y'] (an integer from 0 to 9) is equal to the first column in the data, d[0], in $p$% of cases, and is a random value on the remainder.

*You've fit a classifier to the data, and shown that it's accurate $p$% of the time: it's right when the label matches the first column, and wrong when the label is set at random.

Unfortunately, all this tells you about the classifier is that it works well when one of the predictor columns is identical to the classification label, and badly when the classification label is picked at random. It might be a useful way of showing that an algorithm isn't working properly, but pretty much any properly-programmed classifier should behave in the same way here.
