Validity of AUC for binary categorical variables Scikit-learn function roc_auc_score can be used to get area under curve (AUC) of ROC curve. This score is generally used for numeric predictors' value in predicting outcomes.
However, this function can also be used for categorical variables also. Following is an example (in Python language) where the variable sex is used to predict variable survived and AUC is obtained using this function:
import seaborn, pandas, sklearn
from sklearn.metrics import roc_auc_score
tdf = seaborn.load_dataset('titanic')
print(tdf[['survived','sex']].head(10))
x = tdf['sex'].apply(lambda x: 1 if x=='female' else 0)
y = tdf['survived']
auc = roc_auc_score(y, x)   
auc = round(auc, 4)
print()
print("AUC for sex to predict survived:", auc)

Output:
   survived     sex
0         0    male
1         1  female
2         1  female
3         1  female
4         0    male
5         0    male
6         0    male
7         0    male
8         1  female
9         1  female


AUC for sex to predict survived: 0.7669

However, is this technique statistically sound? Does the AUC obtained using this method a valid value for the relation between 2 categorical variables? Thanks for your help.
Edit: I have reversed the coding of sex to 0 and 1, so that the AUC now is 0.7669
Edit2: From very interesting answers given below, following points seem important:

*

*AUC can be used with categorical variables also, provided it is interpreted correctly.


*It needs to be emphasized that the greater the AUC is away from 0.5, the better it is, and not necessarily higher. Hence, AUC of 0.1 is more predictive, albeit in opposite direction, than AUC of 0.7


*One may report "Absolute AUC" given by following simple Python code:
Abs_AUC = AUC if (AUC>0.5)
else (1-AUC)
Hence, for an AUC of 0.1, the absolute AUC is 0.9;  this will help in comparing AUCs of different variables without missing out ones on the other side of the diagonal in the ROC curve. Note: this is being suggested for predicted variable with only 2 categories.

It's helpful to see that the ROC curve here isn't really a curve.
Instead, you're effectively producing a model that says P(Survive|Male) = .18 and P(Survive|Female) = .74 (the averages in the data), and making predictions using a range of thresholds, e.g. prediction = 1 if p_survive > threshold, or 0 otherwise.
You end up predicting everyone will survive for any threshold < .18, that all females and no males will survive for thresholds between .18 and .74, and that no one will survive with a threshold > .74. This should hopefully make it clear that calculating the AUC or drawing the ROC doesn't really provide any extra information here, since changing the threshold doesn't affect the predictions unless you set it to a daft value. However, it also shows that the AUC score you obtain is still a valid one.


           true_positives  false_positives
threshold                                 
0.0                  1.00             1.00
0.1                  1.00             1.00
0.2                  0.68             0.15
0.3                  0.68             0.15
0.4                  0.68             0.15
0.5                  0.68             0.15
0.6                  0.68             0.15
0.7                  0.68             0.15
0.8                  0.00             0.00
0.9                  0.00             0.00
1.0                  0.00             0.00

Code
p_male, p_female = [tdf.loc[tdf['sex'] == sex, 'survived'].mean() for sex in ['male', 'female']]
tdf['p_survived'] = np.where(tdf['sex'] == 'male', p_male, p_female)

thresholds = np.linspace(0, 1, 11)

def check_calibration(threshold, predicted_probs, outcome):
    prediction = 1 * (predicted_probs > threshold)
    return {
        'true_positives' : prediction[outcome == 1].mean(),
        'false_positives' : prediction[outcome == 0].mean()        
    }

calibration = pd.DataFrame([
    check_calibration(thresh, tdf['p_survived'], tdf['survived'])
    for thresh in thresholds
]).fillna(0)
calibration.index = pd.Index(thresholds, name = 'threshold')

print(calibration.round(2))

accuracies.plot()
plt.xlabel('Threshold (Predict "Survived" if P(Survived > Threshold))')
plt.ylabel('True/False Positive Rate')
plt.title('Calibration')

plt.figure(figsize=(5,5))
plt.plot(accuracies['false_positives'], accuracies['true_positives'])
plt.scatter(accuracies['false_positives'], accuracies['true_positives'])
plt.plot([0,1], [0,1], linestyle = 'dashed', color = 'k')
plt.xlabel('False Positives')
plt.ylabel('True Positives')
plt.title('ROC Curve')

