How to initialize final layer to get a good starting loss? In this post Karpathy says

verify loss @ init. Verify that your loss starts at the correct loss value. E.g. if you initialize your final layer correctly you should measure -log(1/n_classes) on a softmax at initialization. The same default values can be derived for L2 regression, Huber losses, etc.

I understand that having $L = -\log\frac{1}{\text{n classes}} = \log (\text{n classes})$ is the loss you have when you output the same probability for every class when the problem is balanced (ie: the same number of examples for each class). However, I don't understand how can you initialize your network to output this distribution before training.
Do you know what is he referring to with "initialize your final layer correctly"? Which strategy can you follow to force your network to have an initial loss of -log(1/n_classes)?

You could use the Xavier initialization described in this paper. It allows the output of the classification layer (fully connected + softmax activation function) to be close to a uniform distribution, which corresponds to a loss of -log(1/n_classes).
