"Dumb" log-loss for a binary classifier I am trying to understand how I can best compare a classifier that I have trained and tuned against a "dumb" classifier, particularly in the context of binary classification with imbalanced classes.
Here's a summary of my experiment: suppose I have a dataset that I have split in training/validation/test sets.
My classifier (random forest or gradient boosted trees here) is trained on the training set and then the best hyperparameters are selected by evaluating the log-loss on the validation set and choosing the model with the lowest log-loss. I am choosing here to minimise the log-loss because I want the model to incur a very high penalty when it confidently predicts the wrong class.
Suppose now that I want to compare my classifier against a dumb classifier on the test set and suppose that I have the following class proportions in the test set:
55% class 0 and 45% class 1.
Suppose also that the accuracy of my classifier is 60%.
A particularly dumb classifier is one that would for example classify everything as 0, which would result in an accuracy of 55%. This would be of course a poor benchmark to use, even as a dumb baseline.
Say I want to compare my classifier against a random one, for example a biased coin.
How do I calculate the log-loss of the biased coin?
I thought initially that I should use the proportion of classes in the test set as the probabilities of the biased coin i.e.
$$P[X=0]=0.55 \text{ and } P[X=1]=0.45$$
This would result in a log-loss of :
$$-\frac{1}{N}\sum_{i=1}^{N}(y_i\log(p_i) + (1-y_i)\log(1-p_i))=-(0.55\log(0.55) + 0.45\log(0.45))=0.6881$$
In doing so, I feel like I am making a mistake since it would be like this random classifier already knows the distribution of my test set?
Is the correct thing to assume that the probabilities of my biased coin classifier are the ones observed in the training set (for example 51% class 0 and 49% class 1) and simulate randomly the choice of either class 0 or 1 on my test set using those probabilities so as to calculate a log-loss?
Thanks!

You are correct, your if your "dumb" classifier knows the frequency of successes in the test set, it in fact works as an oracle, and is not that dumb. You're leaking the data from test set. It is easy to imagine an extreme case with big discrepancy between train and test set where such "dumb" classifier would in fact outperform the model trained using only the train set.
What you should do is to base your "dumb" classifier on the distribution of the train dataset. In fact, for binary data predicting the mean, or probability of success, is the best single-value prediction you can make assuming squared error or log-loss, so it is a pretty nice benchmark of the simplest but not completely useless model.
